Mgiyuk

Check if an IP address is in a IPNetwork with Pyspark With Pyspark, I would like to join/merge if an IP address in the dataframe A is in a IP network range or hits the same IP address in the dataframe B. The dataframe A contains IP addresses only and the other one has IP addresses or IP addresses with a CIDR. Here is an example. Dataframe A +---------------+ | ip_address| +---------------+ | 192.0.2.2| | 164.42.155.5| | 52.95.245.0| | 66.42.224.235| | ...| +---------------+ Dataframe B +---------------+ | ip_address| +---------------+ | 123.122.213.34| | 41.32.241.2| | 66.42.224.235| | 192.0.2.0/23| | ...| +---------------+ then an expected output is something like below +---------------+--------+ | ip_address| is_in_b| +---------------+--------+ | 192.0.2.2| true| -> This is in the same network range as 192.0.2.0/23 | 164.42.155.5| false| | 52.95.245.0| false| | 66.42.224.235| true| -> This is in B | ...