Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions python/sedona/spark/stats/autocorrelation/moran.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,24 @@

@dataclass
class MoranResult:
"""Result of the global Moran's I test.

Attributes:
i: Moran's I. Values near 1 mean that similar values are close to each other, values near -1
mean that neighbors tend to have dissimilar values, and values near 0 mean that the values
are randomly distributed.
p_norm: p-value of ``z_norm`` under the normality assumption. It is two-tailed unless
``Moran.get_global`` was called with ``two_tailed=False``.
z_norm: z-score of ``i`` under the normality assumption.
"""

i: float
p_norm: float
z_norm: float


class Moran:
"""Global Moran's I spatial autocorrelation test."""

@staticmethod
def get_global(
Expand All @@ -36,6 +48,22 @@ def get_global(
id_column: str = "id",
value_column: str = "value",
) -> MoranResult:
"""Computes the global Moran's I of a value column.

The dataframe needs an id column that uniquely identifies each row, a numeric value column, and a
column named ``weights`` that lists each row's neighbors and their weights, such as the one added by
``sedona.spark.stats.weighting.add_distance_band_column``. Each neighbor must include the id and value
columns, so pass them to the weighting function in ``saved_attributes``.

Args:
df: dataframe with the id, value, and weights columns
two_tailed: whether to return a two-tailed p-value. Default is True
id_column: name of the column that identifies each row. Default is "id"
value_column: name of the numeric column to test. Default is "value"

Returns:
A MoranResult with Moran's I and its z-score and p-value under the normality assumption.
"""
sedona = SparkSession.getActiveSession()

_jvm = sedona._jvm
Expand Down
2 changes: 1 addition & 1 deletion python/sedona/spark/stats/clustering/dbscan.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def dbscan(
min_pts: minimum number of points parameter of DBSCAN algorithm
geometry: name of the geometry column
include_outliers: whether to return outlier points. If True, outliers are returned with a cluster value of -1.
Default is False
Default is True
use_spheroid: whether to use a cartesian or spheroidal distance calculation. Default is false
is_core_column_name: what the name of the column indicating if this is a core point should be. Default is "isCore"
cluster_column_name: what the name of the column indicating the cluster id should be. Default is "cluster"
Expand Down
14 changes: 8 additions & 6 deletions python/sedona/spark/stats/hotspot_detection/getis_ord.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,23 +38,25 @@ def g_local(

Weights should be the neighbors of this row. The members of the weights should be comprised of structs containing a
value column and a neighbor column. The neighbor column should be the contents of the neighbors with the same types
as the parent row (minus neighbors). You can use `wherobots.weighing.add_distance_band_column` to achieve this. To
calculate the Gi* statistic, ensure the focal observation is in the neighbors array (i.e. the row is in the weights
column) and `star=true`. Significance is calculated with a z score. Permutation tests are not yet implemented and
thus island weight does nothing. The following columns will be added: G, E[G], V[G], Z, P.
as the parent row (minus neighbors). You can use `sedona.spark.stats.weighting.add_distance_band_column` to
achieve this. To calculate the Gi* statistic, ensure the focal observation is in the neighbors array (pass
`include_self=True` to the weighting function) and set `star=True`. Significance is calculated with a z score.
Permutation tests are not yet implemented and thus island weight does nothing. The following columns will be
added: G, EG (the expected G), VG (the variance of G), Z, and P (1 minus the standard normal CDF of |Z|).

Args:
dataframe: the dataframe to perform the G statistic on
x: The column name we want to perform hotspot analysis on
weights: The column name containing the neighbors array. The neighbor column should be the contents of
the neighbors with the same types as the parent row (minus neighbors). You can use
`wherobots.weighing.add_distance_band_column` to achieve this.
`sedona.spark.stats.weighting.add_distance_band_column` to achieve this.
permutations: Not used. Permutation tests are not supported yet. The number of permutations to use for the
significance test.
star: Whether the focal observation is in the neighbors array. If true this calculates Gi*, otherwise Gi
island_weight: Not used. The weight for the simulated neighbor used for records without a neighbor in perm tests

Returns:
A dataframe with the original columns plus the columns G, E[G], V[G], Z, P.
A dataframe with the original columns plus the columns G, EG, VG, Z, P.
"""
sedona = SparkSession.getActiveSession()

Expand Down
23 changes: 12 additions & 11 deletions python/sedona/spark/stats/weighting.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,21 +40,22 @@ def add_distance_band_column(
The dataframe should contain at least one GeometryType column. Rows must be unique. If one
geometry column is present it will be used automatically. If two are present, the one named
'geometry' will be used. If more than one are present and neither is named 'geometry', the
column name must be provided. The new column will be named 'cluster'.
column name must be provided. The new column is named by result_name, 'weights' by default.

Args:
dataframe: DataFrame with geometry column
threshold: Distance threshold for considering neighbors
binary: whether to use binary weights or inverse distance weights for neighbors (dist^alpha)
alpha: alpha to use for inverse distance weights ignored when binary is true
include_zero_distance_neighbors: whether to include neighbors that are 0 distance. If 0 distance neighbors are
included and binary is false, values are infinity as per the floating point spec (divide by 0)
included, binary is false, and alpha is negative, their weight is infinite (division by zero)
include_self: whether to include self in the list of neighbors
self_weight: the value to use for the self weight
geometry: name of the geometry column
use_spheroid: whether to use a cartesian or spheroidal distance calculation. Default is false
saved_attributes: the attributes to save in the neighbor column. Default is all columns.
result_name: the name of the resulting column. Default is 'weights'.

Returns:
The input DataFrame with a weight column added containing neighbors and their weights added to each row.

Expand Down Expand Up @@ -93,13 +94,12 @@ def add_binary_distance_band_column(
Weights will always be 1.0. The dataframe should contain at least one GeometryType column. Rows must be unique. If
one geometry column is present it will be used automatically. If two are present, the one named 'geometry' will be
used. If more than one are present and neither is named 'geometry', the column name must be provided. The new column
will be named 'cluster'.
is named by result_name, 'weights' by default.

Args:
dataframe: DataFrame with geometry column
threshold: Distance threshold for considering neighbors
include_zero_distance_neighbors: whether to include neighbors that are 0 distance. If 0 distance neighbors are
included and binary is false, values are infinity as per the floating point spec (divide by 0)
include_zero_distance_neighbors: whether to include neighbors that are 0 distance. Default is True
include_self: whether to include self in the list of neighbors
geometry: name of the geometry column
use_spheroid: whether to use a cartesian or spheroidal distance calculation. Default is false
Expand Down Expand Up @@ -148,14 +148,15 @@ def add_weighted_distance_band_column(
Weights will be distance^alpha. The dataframe should contain at least one GeometryType column. Rows must be unique. If
one geometry column is present it will be used automatically. If two are present, the one named 'geometry' will be
used. If more than one are present and neither is named 'geometry', the column name must be provided. The new column
will be named 'cluster'.
is named by result_name, 'weights' by default.

Args:
dataframe: DataFrame with geometry column
threshold: Distance threshold for considering neighbors
alpha: alpha to use for inverse distance weights. Computation is dist^alpha. Default is -1.0
include_zero_distance_neighbors: whether to include neighbors that are 0 distance. If 0 distance neighbors are
included and binary is false, values are infinity as per the floating point spec (divide by 0)
alpha: exponent of the distance weights, computed as dist^alpha. Use a negative value, such as -1.0, for
inverse distance weights
include_zero_distance_neighbors: whether to include neighbors that are 0 distance. Default is True. With a
negative alpha, their weight is infinite (division by zero)
include_self: whether to include self in the list of neighbors
self_weight: the value to use for the self weight. Default is 1.0
geometry: name of the geometry column
Expand All @@ -164,8 +165,8 @@ def add_weighted_distance_band_column(
result_name: the name of the resulting column. Default is 'weights'.

Returns:
The input DataFrame with a weight column added containing neighbors and their weights (always 1) added to each
row.
The input DataFrame with a weight column added containing neighbors and their weights (dist^alpha) added to
each row.

"""
sedona = SparkSession.getActiveSession()
Expand Down
Loading