bolster.data_sources.justice.mortgages ====================================== .. py:module:: bolster.data_sources.justice.mortgages .. autoapi-nested-parse:: NICTS Mortgages: Action for Possession Data. Provides access to quarterly statistics on mortgage possession proceedings in the Chancery Division of the Northern Ireland High Court, published by the Northern Ireland Courts and Tribunals Service (NICTS). Three datasets are available: - **Cases received** - writs and originating summonses issued (Table 1). - **Cases disposed** - cases concluded by the court (Table 2). - **Final orders** - the type of final order made, e.g. possession, suspended possession, strike out (Table 3, available from 2017 onwards). Data Source: **Publication Page**: https://www.justice-ni.gov.uk/publications/nicts-mortgages-action-possession The module scrapes this page to find the latest quarterly ODS file (``mortgages-bulletin-tables-.ods``), which contains separate worksheets for received, disposed and final-order statistics. Update Frequency: Quarterly Geographic Coverage: Northern Ireland Reference Period: 2007 - present (final orders: 2017 - present) This data pairs well with :mod:`bolster.data_sources.ni_house_price_index` for contextualising the housing market against repossession activity. .. rubric:: Example >>> from bolster.data_sources.justice import mortgages >>> df = mortgages.get_cases_received() >>> "applications" in df.columns True >>> {"Q1", "Q2", "Q3", "Q4"}.issubset(set(df["quarter"])) True Attributes ---------- .. autoapisummary:: bolster.data_sources.justice.mortgages.logger bolster.data_sources.justice.mortgages.PUBLICATION_URL bolster.data_sources.justice.mortgages.BASE_URL bolster.data_sources.justice.mortgages.SHEET_RECEIVED bolster.data_sources.justice.mortgages.SHEET_DISPOSED bolster.data_sources.justice.mortgages.SHEET_FINAL_ORDERS bolster.data_sources.justice.mortgages.QUARTERS Exceptions ---------- .. autoapisummary:: bolster.data_sources.justice.mortgages.MortgagesDataError bolster.data_sources.justice.mortgages.MortgagesDataNotFoundError bolster.data_sources.justice.mortgages.MortgagesValidationError Functions --------- .. autoapisummary:: bolster.data_sources.justice.mortgages.get_latest_publication_url bolster.data_sources.justice.mortgages.download_file bolster.data_sources.justice.mortgages.parse_data bolster.data_sources.justice.mortgages.get_latest_data bolster.data_sources.justice.mortgages.get_cases_received bolster.data_sources.justice.mortgages.get_cases_disposed bolster.data_sources.justice.mortgages.get_final_orders bolster.data_sources.justice.mortgages.validate_data bolster.data_sources.justice.mortgages.clear_cache Module Contents --------------- .. py:data:: logger .. py:data:: PUBLICATION_URL :value: 'https://www.justice-ni.gov.uk/publications/nicts-mortgages-action-possession' .. py:data:: BASE_URL :value: 'https://www.justice-ni.gov.uk' .. py:data:: SHEET_RECEIVED :value: 'Mortgages_received' .. py:data:: SHEET_DISPOSED :value: 'Mortgages_disposed' .. py:data:: SHEET_FINAL_ORDERS :value: 'Mortgages_final_orders' .. py:data:: QUARTERS :value: ['Q1', 'Q2', 'Q3', 'Q4'] .. py:exception:: MortgagesDataError Bases: :py:obj:`Exception` Base exception for NICTS mortgages data errors. Initialize self. See help(type(self)) for accurate signature. .. py:exception:: MortgagesDataNotFoundError Bases: :py:obj:`MortgagesDataError` Raised when the data file cannot be located or downloaded. Initialize self. See help(type(self)) for accurate signature. .. py:exception:: MortgagesValidationError Bases: :py:obj:`MortgagesDataError` Raised when downloaded data fails validation. Initialize self. See help(type(self)) for accurate signature. .. py:function:: get_latest_publication_url(base_url = PUBLICATION_URL) Find the URL of the most recent mortgages bulletin ODS file. Scrapes the publication landing page for links to ``.ods`` files and returns the first one found (the page lists newest first). :param base_url: URL of the publication listing page. :returns: Absolute URL of the latest ODS bulletin file. :raises MortgagesDataNotFoundError: If the page cannot be fetched or no ODS link is found. .. rubric:: Example >>> url = get_latest_publication_url() # doctest: +SKIP >>> url.endswith(".ods") # doctest: +SKIP True .. py:function:: download_file(url, cache_ttl_hours = 24 * 7, force_refresh = False) Download a bulletin ODS file with caching. :param url: URL of the ODS file to download. :param cache_ttl_hours: Cache validity in hours (default: 7 days, since data is published quarterly). :param force_refresh: If True, bypass the cache and re-download. :returns: Path to the downloaded (or cached) file. :raises MortgagesDataNotFoundError: If the download fails. .. py:function:: parse_data(file_path) Parse all mortgage tables from a bulletin ODS file. :param file_path: Path to the ODS bulletin file. :returns: Dictionary with keys ``received``, ``disposed`` and ``final_orders`` mapping to tidy long-format DataFrames. ``final_orders`` may be absent if the file pre-dates 2017. .. rubric:: Example >>> tables = parse_data(download_file(get_latest_publication_url())) # doctest: +SKIP >>> sorted(tables) # doctest: +SKIP ['disposed', 'final_orders', 'received'] .. py:function:: get_latest_data(force_refresh = False) Download and parse the latest mortgages bulletin. :param force_refresh: If True, bypass the cache and download fresh data. :returns: Dictionary of tidy DataFrames keyed ``received``, ``disposed`` and (where available) ``final_orders``. .. rubric:: Example >>> data = get_latest_data() >>> "received" in data and "disposed" in data True .. py:function:: get_cases_received(force_refresh = False) Get quarterly mortgage possession cases received (Table 1). :param force_refresh: If True, bypass the cache and download fresh data. :returns: year, quarter, period, applications, annual_total, annual_pct_change. One row per (year, quarter) from 2007. :rtype: DataFrame with columns .. rubric:: Example >>> df = get_cases_received() >>> "applications" in df.columns True .. py:function:: get_cases_disposed(force_refresh = False) Get quarterly mortgage possession cases disposed (Table 2). :param force_refresh: If True, bypass the cache and download fresh data. :returns: year, quarter, period, applications, annual_total, annual_pct_change. One row per (year, quarter) from 2007. :rtype: DataFrame with columns .. rubric:: Example >>> df = get_cases_disposed() >>> "applications" in df.columns True .. py:function:: get_final_orders(force_refresh = False) Get mortgage possession final orders by type (Table 3). Final orders have been published from 2017 onwards. Earlier years are reported annually; from 2025 they are broken down by quarter. :param force_refresh: If True, bypass the cache and download fresh data. :returns: order_type, year, quarter, period, count. :rtype: DataFrame with columns :raises MortgagesDataNotFoundError: If the bulletin does not include final orders (e.g. very old files). .. rubric:: Example >>> df = get_final_orders() # doctest: +SKIP >>> "order_type" in df.columns # doctest: +SKIP True .. py:function:: validate_data(df, value_col = 'applications', min_records = 40) Validate a parsed quarterly mortgages DataFrame. Checks structure and sanity of received/disposed tables: - Required columns are present. - There are at least ``min_records`` rows (>= 10 years of quarters). - Years fall within a plausible range (2007 onwards). - All non-null counts are non-negative. :param df: DataFrame to validate (received or disposed). :param value_col: Name of the count column to check (default: "applications"). :param min_records: Minimum acceptable number of rows. :returns: True if the data passes all checks. :raises MortgagesValidationError: If any validation check fails. .. rubric:: Example >>> import pandas as pd >>> validate_data(pd.DataFrame()) Traceback (most recent call last): ... bolster.data_sources.justice.mortgages.MortgagesValidationError: DataFrame is empty .. py:function:: clear_cache() Clear all cached mortgages bulletin files. :returns: Number of files deleted.