Mgiyuk

Spark writing to Cassandra with varying TTL In Java Spark, I have a dataframe that has a 'bucket_timestamp' column, which represents the time of the bucket that the row belongs to. I want to write the dataframe to a Cassandra DB. The data must be written to the DB with TTL. The TTL should be depended on the bucket timestamp - where each row's TTL should be calculated as ROW_TTL = CONST_TTL - (CurrentTime - bucket_timestamp) , where CONST_TTL is a constant TTL that I configured. ROW_TTL = CONST_TTL - (CurrentTime - bucket_timestamp) CONST_TTL Currently I am writing to Cassandra with spark using a constant TTL, with the following code: df.write().format("org.apache.spark.sql.cassandra") .options(new HashMap<String, String>() { { put("keyspace", "key_space_name"); put("table, "table_name"); put("spark.cassandra.output.ttl, Long.toStri...

Storing counter inside cassandra collection I want to store aggregation data from my sensor, here's my schema plan for cassandra table UPDATED CREATE TABLE every_second_aggregate_signature( device_id text, year int, month int, day int, hour int, minute int, second int, signature map<text,counter>, PRIMARY KEY ((device_id, year, month, day, hour, minute, second)) )WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC, minute DESC, second DESC); The signature data is in form of increment value and dynamic value, ex. ts1 - {"4123sad" : 80, "djas10" : 99} ts2 - {"4123sad" : 83, "djas10" : 103} ts3 - {"4123sad" : 87, "djas10" : 198, "asfac9" : 281} ts4 - {"4123sad" : 89, "djas10" : 201, "asfac9" : 540, "asd81" : 12} The problem is I knew that cassandra didn't support counter inside the collection. Are there any alterna...