Mgiyuk

Best strategy for repartionBy with few big partitions I have to repartition geo data by quadkey. Primarily all the data is pretty balanced, but few partitions are 500x times bigger than others. So it causes very unbalanced partition stage, like 20-30 of 3500 tasks are 98 % slower than others. Is there are any good strategy in that case? I need to do next: stage.repartition(partitionColumns.map(new org.apache.spark.sql.Column(_)):_*) .write.partitionBy(partitionColumns:_*) .format("parquet") .option("compression", "gzip") .mode(SaveMode.Append) .save(destUrl) 1 Answer 1 The .repartition is unnecessary and is probably causing the issue. .repartition If you leave that out and just have the .write.partitionBy... , you will still get the same directory structure, you will just have multiple files within each directory. .write.partitionBy... ...

Pick-up script on Player or Coin game object I watched Basic Platformer Game tutorial for Unity where presenter created coin pick-up script that he attached to Coin prefab. Let's say we have a game with pick-upable objects. They do nothing more than incrementing the score (or affecting the player in another way) and destroy themselves on collision. I was wondering that what is the preferred approach to this problem. I've come up with two approaches: Approach A Have one ObjectPickup script on the player game object . This script would do whatever is required depending on the type of collided object. private void OnTriggerEnter2D(Collider2D other) { if (other.gameObject.CompareTag("Coin")) { IncrementScore(); Destroy(other.gameObject); } else if (other.gameObject.CompareTag("SuperSpeed")) { IncreasePlayerSpeed(); Destroy(other.gameObject); } } Approach B Have CoinPickup script on every coin and SuperSp...

Julia loops are as slow as R loops The code below in Julia and R is to show that the estimator of the population variance is a biased estimator, that is it depends on the sample size and no matter how many times we average over different observations, for small number of data points it is not equal to the variance of the population. It takes for Julia ~10 seconds to finish the two loops and R does it in ~7 seconds. If I leave the code inside the loops commented then the loops in R and Julia take the same time and if I only sum the iterators by s = s + i+ j Julia finishes in ~0.15s and R in ~0.5s. s = s + i+ j Is it that Julia loops are slow or R became fast? How can I improve the speed of the code below for Julia? Can the R code become faster? Julia: using Plots trials = 100000 sample_size = 10; sd = Array{Float64}(trials,sample_size-1) tic() for i = 2:sample_size for j = 1:trials res = randn(i) sd[j,i-1] = (1/(i))*(sum(res.^2))-(1/((i)*i))*(sum(res)*sum(res)) ...

Most effective and efficient way to store tree with co-indexed nodes in XML I am picking up an older project of mine where effectiveness and efficiency are key. I have 100's of GB of XML data that I parse. For each XML tree (millions of them) some XML attributes are used from which patterns are deducted. For this question, though, I shall simplify things greatly - but it is important to remember that there is a lot of data and that fast processing, and tidy storing of the results in XML is important. In addition, the resulting XML tree will need to be traversed as well. In fact, it will serve as a custom indexing mechanism used in BaseX but I'll come back to that later on. From every tree (and its subtrees, but that's not important now) a pattern is created that is based on the root node's direct children. As a basic example, take the following XML tree: <node letter="X"> <node letter="A"/> <node letter="B"/> <no...

Add / substract between matrix and vector in pytorch I want to do + / - / * between matrix and vector in pytorch. How can I do with good performance? I tried to use expand, but it's really slow (I am using big matrix with small vector). a = torch.rand(2,3) print(a) 0.7420 0.2990 0.3896 0.0715 0.6719 0.0602 [torch.FloatTensor of size 2x3] b = torch.rand(2) print(b) 0.3773 0.6757 [torch.FloatTensor of size 2] a.add(b) Traceback (most recent call last): File "C:ProgramDataAnaconda3libsite-packagesIPythoncoreinteractiveshell.py", line 3066, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-17-a1cb1b03d031>", line 1, in <module> a.add(b) RuntimeError: inconsistent tensor size, expected r_ [2 x 3], t [2 x 3] and src [2] to have the same number of elements, but got 6, 6 and 2 elements respectively at c:miniconda2conda-bldpytorch-cpu_1519449358620worktorchlibthgeneric/THTensorMath.c:1021 Expected result: ...

Make matrix from two columns of a dataframe and populate it by third without nested for loop Let's say I have a dataframe with three of its columns being > df A B C 1232 27.3 0.42 1232 27.3 0.36 1232 13.1 0.15 7564 13.1 0.09 7564 13.1 0.63 The required output is: [1232] [7564] [13.1] 0.15 0.36 [27.3] 0.39 0 I need to make a matrix with unique values in A and B as my rows and columns. The value for any cell in the matrix is to be calculated by subsetting the original dataframe for the particular value of A and B and calculating the mean of column C. My code is: mat <- matrix(rep(0), length(unique(df$A)), nrow = length(sort(unique(df$B)))) # sort is to avoid NA colnames(mat) <- unique(df$A) rownames(mat) <- unique(df$B) for (row in rownames(mat)) { for (col in colnames(mat)) { x <- subset(df, A == col & B == row) mat[row, col] = mean(df$C) } } This is very slow, considering I have to ...

Efficient way to work with 2D Arrays (multiple formats) I have an efficiency/performance question in Python: I am planning to store data in some kind of matrix format in a "storage" For example: elementID - nodeID - elementType - Value My current process is to loop through the raw data information and check if there is already an entry with "elementID, nodeID, elementType" in the storage. Now, when considering a second case, also a second value is occuring. What I want to do is now to add the second value to the associated case in the storage. So: elementID - nodeID - elementType - Value1 - Value2 Currently I am working with 2D arrays, so [[a,b,c],[b,c,d],....]. Since this is implying "for loops", the time required to check if the entry is already in the storage is increasing tremendously. Also to add the additional column to the storage is requiring the search for the corresponding entry. In some cases, it is also possible that there are many value column...

counting number of each substring in array python I have a string array for example [a_text, b_text, ab_text, a_text] . I would like to get the number of objects that contain each prefix such as ['a_', 'b_', 'ab_'] so the number of 'a_' objects would be 2. string [a_text, b_text, ab_text, a_text] ['a_', 'b_', 'ab_'] 'a_' so far I've been counting each by filtering the array e.g num_a = len(filter(lambda x: x.startswith('a_'), array)) . I'm not sure if this is slower than looping through all the fields and incrementing each counter since I am filtering the array for each prefix I am counting. Are functions such as filter() faster than a for loop? For this scenario I don't need to build the filtered list if I use a for loop so that may make it faster. num_a = len(filter(lambda x: x.startswith('a_'), array)) filter() Also perhaps instead of the filter I could use list comprehension to make it ...