Skip to main content

Best Practices for Handling Large Spatial-Temporal Data in Shiny Apps

Latoya
  • Updated

Description

When working with large spatial-temporal datasets in Shiny applications, choosing the right file format and implementing proper data loading strategies is crucial for optimal performance and reliability.

File Format Considerations

NetCDF (.nc) Format:

  • Designed specifically for multidimensional spatial-temporal data.
  • Supports rich metadata and compression.
  • Enables efficient subsetting by dimension (time, depth, location).
  • Works well with packages like stars and terra for lazy loading.
  • Only loads required data slices, reducing I/O overhead.

RDS (.rds) Format:

  • Fast and simple for storing R-native objects.
  • Does not support selective reads.
  • Loading multiple small files can increase I/O overhead.
  • Better suited for smaller datasets or when full data loading is required.

Best Practices for Cloud Storage Access

When accessing data from cloud storage services (like Google Cloud Storage) in Shiny apps, consider these strategies to improve reliability:

1. Implement Retry Logic

  • Use error handling and retry mechanisms for downloads.
  • Implement exponential backoff to handle temporary failures.
  • Consider using packages like 'purrr' for safe downloads.

2. Implement Caching

  • Cache downloaded data to avoid repeated requests.
  • Use memoise or reactiveVal for session-level caching.
  • Consider implementing disk caching for frequently accessed files.

3. Optimize Data Structure

  • Combine smaller files into larger chunks (e.g., monthly or seasonal).
  • Use appropriate compression methods.
  • Structure data to allow efficient subsetting.

4. Performance Optimization

  • Use lazy loading when possible.
  • Implement progressive loading for large datasets.
  • Consider using packages optimized for spatial data like 'stars' or 'terra'.

  • Monitor memory usage and implement cleanup strategies.

     

Helpful links

 

While direct code support is outside of our SLA, if you're still experiencing issues or have questions about best practices, you're welcome to reach out to Support by opening a ticket here.

Related articles