Mgiyuk

Is using function in transformation causing Not Serializable exceptions? I have a Breeze DenseMatrix , i find mean per row and mean of squares per row and put them in another DenseMatrix , one per column. But i get Task Not Serializable exception. I know that sc is not Serializable but i think that the exception is because i call functions in a transformation in Safe Zones. Breeze DenseMatrix mean mean DenseMatrix Task Not Serializable sc Serializable Am i right? And how could be a possible way to be done without any functions? Any help would be great! Code: object MotitorDetection { case class MonDetect() extends Serializable { var sc: SparkContext = _ var machines: Int=0 var counters: Int=0 var GlobalVec= BDM.zeros[Double](counters, 2) def findMean(a: BDM[Double]): BDV[Double] = { var c = mean(a(*, ::)) c} def toMatrix(x: BDV[Double], y: BDV[Double], C: Int): BDM[Double]={ val m = BDM.zeros[Double](C,2) m(::, 0) := x m(::, 1) := y m} def SafeZones(stream: DStream[(In...

scala type class with higher kinded types and variance I have a question very similar to this one: Scala higher kinded type variance This, however is slightly different in that, well, it doesn't compile (scala 2.11.8). The basic idea is to take a provided array of "things". If the array is null, return a default value of some type (e.g. Boolean , Option , List[Int] ), otherwise do work on the array and produce a result. The result and default value have the same type. Boolean Option List[Int] The challenge I'm having is getting this to work across a broad set of result types. Here's a contrived example: trait NullGuard[F[_]] { def nullGuard[A, B](arr: Array[A], default: F[B])(expr: => F[B]): F[B] = if (arr == null || arr.length == 0) default else expr } Let's create an implementation that returns an Option: implicit def optionNullGuard[F[X] <: Option[X]]: NullGuard[F] = new NullGuard[F]() {} The above does compile, but the following att...

Scala: Frequency of characters in a list I am a novice programmer from a python/java background, who is trying to get to grips with functional programming in scala. I am trying to write a function that will take a list of characters and return a list of their frequencies. For example, List('a','b','a') should return List(('a', 2), ('b', 1)). I am new to scala and tail recursion, so please don't presume much prior knowledge :) Here is what I have: def times(chars: List[Char]): List[(Char, Int)] = { def loop(chars: List[Char], list: List[(Char, Int)]): List[(Char, Int)] = { if (chars.isEmpty) list else { val head = chars.head val freq = chars.count(x=> x == head) (head,freq) :: list loop(chars.tail.filterNot(x=> x == head),list) } } loop(chars,List()) } If I run times(List('a','b','a')) I just get an empty list. Any help would be greatly appreciated! ...

Best strategy for repartionBy with few big partitions I have to repartition geo data by quadkey. Primarily all the data is pretty balanced, but few partitions are 500x times bigger than others. So it causes very unbalanced partition stage, like 20-30 of 3500 tasks are 98 % slower than others. Is there are any good strategy in that case? I need to do next: stage.repartition(partitionColumns.map(new org.apache.spark.sql.Column(_)):_*) .write.partitionBy(partitionColumns:_*) .format("parquet") .option("compression", "gzip") .mode(SaveMode.Append) .save(destUrl) 1 Answer 1 The .repartition is unnecessary and is probably causing the issue. .repartition If you leave that out and just have the .write.partitionBy... , you will still get the same directory structure, you will just have multiple files within each directory. .write.partitionBy... ...

How to save a websocket client's connection and send it later with akka-streams and akka-http I'm trying to follow this part of the akka-http documentation where it talks about handling web socket messages asynchronously akka-http What I am trying to do is this: Here is the code I have so far def hello: Route = { val amt = 1000 val helloRoute: Route = pathPrefix(Constants.apiVersion) { path("hello") { val source: Source[Message, SourceQueueWithComplete[Message]] = { Source.queue(1, OverflowStrategy.backpressure) } val paymentRequest = createPaymentRequest(1000, extractUpgradeToWebSocket) Directives.handleWebSocketMessages( paymentFlow(paymentRequest) ) } } helloRoute } private def createPaymentRequest(amt: Long, wsUpgrade: Directive1[UpgradeToWebSocket]) = { val httpResponse: Directive1[HttpResponse] = wsUpgrade.map { ws => val sink: Sink[Message, NotUsed] = Sink.cancelled() val source: Source[Message, ...

How to convert json file to List using scala I have the following json input sample.json [{"level": 1, "firstFile": "one", "secondFile": "first"}, {"level": 1, "firstFile": "two", "secondFile": "sec"}, {"level": 2, "firstFile": "three", "secondFile": "third"}] i want my result should be: val first= List(List("one","two"),List("three")) val second= List(List("first","sec"),List("third")) build.sbt: name := "Test" version := "1.0" scalaVersion := "2.11.0" libraryDependencies ++= Seq("org.apache.spark" % "spark-sql_2.11" % "2.0.0" % "provided") libraryDependencies += "com.typesafe" % "config" % "1.2.0" libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % ...

How to use Either in scala I want to do similar like below def run(list : List(String)){ val map = Map[String,String] (immutable map) list.foreach{ val x = someOtherMethod() val result = processMap(x,map) // please see below method for details //Not sure how to use result here 1) if ProcessMap returns Map then I want to assign returned MAP to map val 2) otherwise don't do anything } writeOutput(x,map) } def processMap(x :boolean, map : Map[String,String]) : Either[Option(String) , Map[String,String]] { x match { case true => { do some processing and update map and return Right(map ++ newGeneratedMap) } case _ => Left(None) } } In above example I want to use Map as Java static variable. I want to pass Map to a method based on some condition it may get updated and may not. If its updated then only I want to return it and pass it to next method.If map is not updated then I want to pass existing map to method. Main Question : How can I access output of result (left ...

Serialize Guava's MinMaxPriorityQueue After a few days researching why my Flink application is not working properly I've came to the conclusion that the problem resides in a MinMaxPriorityQueue I am using. MinMaxPriorityQueue It seems that this structure is not serializable. I've tried several ways to serialize it: env.getConfig.registerTypeWithKryoSerializer(classOf[MinMaxPriorityQueue[Double]], classOf[JavaSerializer]) env.getConfig.registerTypeWithKryoSerializer(classOf[MinMaxPriorityQueue[java.lang.Double]], classOf[ProtobufSerializer]); env.getConfig().addDefaultKryoSerializer(MyCustomType.class, TBaseSerializer.class); all of them without luck. However I've found this: Serializing Guava's ImmutableTable Is there an equivalent to MinMaxPriorityQueue, or a way to serialize it? Update I've translated Tomasz into scala: class MinMaxPriorityQueueSerializer extends Serializer[MinMaxPriorityQueue[Object]] { private[this] val log = LoggerFactory.getLogger(this....

Create Range with inclusive end value when stepping Is there any way to create a range which includes the end value when using a step which doesn't align? For instance the following yields: scala> Range.inclusive(0, 35, 10) res3: scala.collection.immutable.Range.Inclusive = Range(0, 10, 20, 30) But I would also like the end value (35) included like so: scala> Range.inclusive(0, 35, 10) res3: scala.collection.immutable.Range.Inclusive = Range(0, 10, 20, 30, 35) 3 Answers 3 No, not with the current definition/ implementation. It would be strange behaviour to have the step the same for all intermediate elements but different from the last. Yes that makes sense - however I was considering a use case where a collection (table) could be effectively partitioned into chunks using column indexes. thanks. – Fred Smith May 25 '16 at...