This would require a big shuffle which consequently could cause an OOM exception. Why is that? Well you could indeed use pivot as brilliantly already implemented although I believe that that wouldn't work when the cardinality of the query parameters is very high i.e 500 or more unique params.
The basic idea here is that we divide the problem into two sub-problems, first get all the possible query parameters and then we extract the values for all the urls. Here we return the second value since the first one will be the query parameter name. : Finally split by = to retrieve the value of the query parameter. For the first parameter this will return param1=1, for the second param2=a etc. filter will return an array hence we just access the first item since we know that the particular param should always exists. qp stands for the items of the params array. UDF solution: from urllib.parse import urlparse, parse_qsįrom import MapType, StringTypeĮxtract_params = udf(lambda x: =% the param name i.e param1=%.
The former solution uses regex which is a finicky method of extracting parameters from strings but the latter would need to be wrapped in a UDF to be used.
I can think of two possible methods of doing this, using functions.regexp_extract from the pyspark library or by using _qs and from the standard library. What would be the best way of extracting the URL parameters from this column and adding them as columns to the dataframe to produce the below? +-+-+ Say I have a column filled with URLs like in the following: +-+
0 kommentar(er)
