Compare commits

...

7 Commits

Author SHA1 Message Date
stopflock
56d55bb922 bump version 2025-12-07 11:37:56 -06:00
stopflock
d665db868a devibe changelog 2025-12-07 11:37:18 -06:00
stopflock
b0d2ae22fe Simplify suspected locations databse handling 2025-12-07 11:34:38 -06:00
stopflock
ffec43495b Better suspected locations download indicator 2025-12-07 11:00:42 -06:00
stopflock
16b8acad3a Suspected locations database 2025-12-07 10:23:36 -06:00
stopflock
4fba26ff55 Route distance timeout warning 2025-12-06 15:07:34 -06:00
stopflock
b02623deac rework one-time migrations 2025-12-06 14:48:21 -06:00
20 changed files with 1423 additions and 328 deletions

View File

@@ -399,24 +399,53 @@ Users often want to follow their location while keeping the map oriented north.
**Why the change:**
The previous approach tracked both tile loading and surveillance data, creating redundancy since tiles already show loading progress visually on the map. Users don't need to be notified about tile loading issues when they can see tiles loading/failing directly. Focusing only on surveillance data makes the indicator more purposeful and less noisy.
### 11. Suspected Locations
### 11. Suspected Locations (v1.8.0+: SQLite Database Storage)
**Data pipeline:**
- **CSV ingestion**: Downloads utility permit data from alprwatch.org
- **CSV ingestion**: Downloads utility permit data from alprwatch.org (100MB+ datasets)
- **SQLite storage**: Batch insertion into database with geographic indexing (v1.8.0+)
- **Dynamic field parsing**: Stores all CSV columns (except `location` and `ticket_no`) for flexible display
- **GeoJSON processing**: Handles Point, Polygon, and MultiPolygon geometries
- **Proximity filtering**: Hides suspected locations near confirmed devices
- **Regional availability**: Currently select locations, expanding regularly
**Storage architecture (v1.8.0+):**
- **Database**: SQLite with spatial indexing for efficient geographic queries
- **Hybrid caching**: Sync cache for immediate UI response + async database queries
- **Memory efficiency**: No longer loads entire dataset into memory
- **Legacy migration**: Automatic migration from SharedPreferences to SQLite
**Performance improvements:**
- **Startup time**: Reduced from 5-15 seconds to <1 second
- **Memory usage**: Reduced from 200-400MB to <10MB
- **Query time**: Reduced from 100-500ms to 10-50ms with indexed queries
- **Progressive loading**: UI shows cached results immediately, updates with fresh data
**Display approach:**
- **Required fields**: `ticket_no` (for heading) and `location` (for map positioning)
- **Dynamic display**: All other CSV fields shown automatically, no hardcoded field list
- **Server control**: Field names and content controlled server-side via CSV headers
- **Brutalist rendering**: Fields displayed as-is from CSV, empty fields hidden
**Database schema:**
```sql
CREATE TABLE suspected_locations (
ticket_no TEXT PRIMARY KEY,
centroid_lat REAL NOT NULL,
centroid_lng REAL NOT NULL,
bounds TEXT,
geo_json TEXT,
all_fields TEXT NOT NULL
);
CREATE INDEX idx_centroid ON suspected_locations (centroid_lat, centroid_lng);
```
**Why utility permits:**
Utility companies often must file permits when installing surveillance infrastructure. This creates a paper trail that can indicate potential surveillance sites before devices are confirmed through direct observation.
**Why SQLite migration:**
The original SharedPreferences approach became untenable as the CSV dataset grew beyond 100MB, causing memory pressure and long startup times. SQLite provides efficient storage and querying while maintaining the simple, brutalist architecture the project follows.
### 12. Upload Mode Simplification
**Release vs Debug builds:**
@@ -484,11 +513,13 @@ The major performance issue was discovered to be double caching with expensive o
- **Smart queue management**: Drops oldest requests when queue fills up
- **Reduced concurrent connections**: 8 threads instead of 10 for better stability across platforms
### 14. Navigation & Routing (Implemented, Awaiting Integration)
### 14. Navigation & Routing (Implemented and Active)
**Current state:**
- **Search functionality**: Fully implemented and active
- **Avoidance routing**: Fully implemented and active
- **Distance feedback**: Shows real-time distance when selecting second route point
- **Long distance warnings**: Alerts users when routes may timeout (configurable threshold)
- **Offline routing**: Requires vector map tiles
**Architecture:**
@@ -496,6 +527,12 @@ The major performance issue was discovered to be double caching with expensive o
- RoutingService handles API communication and route calculation
- SearchService provides location lookup and geocoding
**Distance warning system (v1.7.0):**
- **Real-time distance display**: Shows distance from first to second point during selection
- **Configurable threshold**: `kNavigationDistanceWarningThreshold` in dev_config (default 30km)
- **User feedback**: Warning message about potential timeouts for long routes
- **Brutalist approach**: Simple distance calculation using existing `Distance()` utility
---
## Key Design Decisions & Rationales

108
V1.8.0_CHANGES_SUMMARY.md Normal file
View File

@@ -0,0 +1,108 @@
# v1.8.0 Changes Summary: Suspected Locations Database Migration
## Problem Solved
The CSV file containing suspected surveillance locations from alprwatch.org has grown beyond 100MB, causing significant performance issues:
- Long app startup times when the feature was enabled
- Memory pressure from loading entire CSV into memory
- Slow suspected location queries due to in-memory iteration
## Solution: SQLite Database Migration
### Brutalist Approach
Following the project's "brutalist code" philosophy, we chose SQLite as the simplest, most reliable solution:
- **Simple**: Well-understood, stable technology
- **Efficient**: Proper indexing for geographic queries
- **Cross-platform**: Works consistently on iOS and Android
- **No cleverness**: Straightforward database operations
### Key Changes
#### 1. New Database Service (`SuspectedLocationDatabase`)
- SQLite database with proper geographic indexing
- Batch insertion for handling large datasets
- Efficient bounds queries without loading full dataset
- Automatic database migration and cleanup
#### 2. Hybrid Caching System (`SuspectedLocationCache`)
- **Async caching**: Background database queries with proper notification
- **Sync caching**: Immediate response for UI with async fetch trigger
- **Smart memory management**: Limited cache sizes to prevent memory issues
- **Progressive loading**: UI shows empty initially, updates when data loads
#### 3. API Compatibility
- Maintained existing API surface for minimal UI changes
- Added sync versions of methods for immediate UI responsiveness
- Async methods for complete data fetching where appropriate
#### 4. Migration Support
- Automatic migration of existing SharedPreferences-based data
- Clean legacy data cleanup after successful migration
- Graceful fallback if migration fails
#### 5. Updated Dependencies
- Added `sqflite: ^2.4.1` for SQLite support
- Added explicit `path: ^1.8.3` dependency
### Database Schema
```sql
-- Main suspected locations table
CREATE TABLE suspected_locations (
ticket_no TEXT PRIMARY KEY, -- Unique identifier
centroid_lat REAL NOT NULL, -- Latitude for spatial queries
centroid_lng REAL NOT NULL, -- Longitude for spatial queries
bounds TEXT, -- JSON array of boundary points
geo_json TEXT, -- Original GeoJSON geometry
all_fields TEXT NOT NULL -- All other CSV fields as JSON
);
-- Spatial index for efficient bounds queries
CREATE INDEX idx_centroid ON suspected_locations (centroid_lat, centroid_lng);
-- Metadata table for tracking fetch times
CREATE TABLE metadata (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
```
### Performance Improvements
#### Before (v1.7.0 and earlier):
- **Startup**: 5-15 seconds to load 100MB+ CSV into memory
- **Memory usage**: 200-400MB for suspected location data
- **Query time**: 100-500ms to iterate through all entries
- **Storage**: SharedPreferences JSON (slower serialization)
#### After (v1.8.0):
- **Startup**: <1 second (database already optimized)
- **Memory usage**: <10MB for suspected location data
- **Query time**: 10-50ms with indexed geographic queries
- **Storage**: SQLite with proper indexing
### UI Changes
- **Minimal**: Existing UI largely unchanged
- **Progressive loading**: Suspected locations appear as data becomes available
- **Settings**: Last fetch time now loads asynchronously (converted to StatefulWidget)
- **Error handling**: Better error recovery and user feedback
### Migration Process
1. **Startup detection**: Check for legacy SharedPreferences data
2. **Data conversion**: Parse legacy format into raw CSV data
3. **Database insertion**: Use new batch insertion process
4. **Cleanup**: Remove legacy data after successful migration
5. **Graceful failure**: Migration errors don't break the app
### Testing Notes
- **No data loss**: Existing users' suspected location data is preserved
- **Backward compatibility**: Users can safely downgrade if needed (will re-fetch data)
- **Fresh installs**: New users get optimal database storage from start
- **Legacy cleanup**: Old storage is automatically cleaned up after migration
### Code Quality
- **Error handling**: Comprehensive try-catch with meaningful debug output
- **Memory management**: Bounded cache sizes, efficient batch processing
- **Async safety**: Proper `mounted` checks and state management
- **Debug logging**: Detailed progress tracking for troubleshooting
This change follows the project's brutalist philosophy: solving the real problem (performance) with the simplest reliable solution (SQLite), avoiding clever optimizations in favor of well-understood, maintainable code.

View File

@@ -1,8 +1,20 @@
{
"1.8.0": {
"content": [
"• Better performance and reduced memory usage when using suspected location data by using a database"
]
},
"1.7.0": {
"content": [
"• Distance display when selecting second navigation point; shows distance from first location in real-time",
"• Long distance warning; routes over 20km display a warning about potential timeouts"
]
},
"1.6.3": {
"content": [
"• Fixed navigation sheet button flow - route to/from buttons no longer reappear after selecting second location",
"• Added cancel button when selecting second route point for easier exit from route planning"
"• Added cancel button when selecting second route point for easier exit from route planning",
"• Removed placeholder FOV values from built-in device profiles - oops"
]
},
"1.6.2": {

View File

@@ -114,6 +114,8 @@ class AppState extends ChangeNotifier {
bool get settingRouteStart => _navigationState.settingRouteStart;
bool get isSettingSecondPoint => _navigationState.isSettingSecondPoint;
bool get areRoutePointsTooClose => _navigationState.areRoutePointsTooClose;
double? get distanceFromFirstPoint => _navigationState.distanceFromFirstPoint;
bool get distanceExceedsWarningThreshold => _navigationState.distanceExceedsWarningThreshold;
bool get isCalculating => _navigationState.isCalculating;
bool get showingOverview => _navigationState.showingOverview;
String? get routingError => _navigationState.routingError;
@@ -173,7 +175,8 @@ class AppState extends ChangeNotifier {
SuspectedLocation? get selectedSuspectedLocation => _suspectedLocationState.selectedLocation;
bool get suspectedLocationsEnabled => _suspectedLocationState.isEnabled;
bool get suspectedLocationsLoading => _suspectedLocationState.isLoading;
DateTime? get suspectedLocationsLastFetch => _suspectedLocationState.lastFetchTime;
double? get suspectedLocationsDownloadProgress => _suspectedLocationState.downloadProgress;
Future<DateTime?> get suspectedLocationsLastFetch => _suspectedLocationState.lastFetchTime;
void _onStateChanged() {
notifyListeners();
@@ -633,13 +636,7 @@ class AppState extends ChangeNotifier {
await _settingsState.setNetworkStatusIndicatorEnabled(enabled);
}
/// Migrate upload queue to new two-stage changeset system (v1.5.3)
Future<void> migrateUploadQueueToTwoStageSystem() async {
// Migration is handled automatically in PendingUpload.fromJson via _migrateFromLegacyFields
// This method triggers a queue reload to apply migrations
await _uploadQueueState.reloadQueue();
debugPrint('[AppState] Upload queue migration completed');
}
/// Set suspected location minimum distance from real nodes
Future<void> setSuspectedLocationMinDistance(int distance) async {
@@ -665,6 +662,11 @@ class AppState extends ChangeNotifier {
_startUploader(); // resume uploader if not busy
}
/// Reload upload queue from storage (for migration purposes)
Future<void> reloadUploadQueue() async {
await _uploadQueueState.reloadQueue();
}
// ---------- Suspected Location Methods ----------
Future<void> setSuspectedLocationsEnabled(bool enabled) async {
await _suspectedLocationState.setEnabled(enabled);
@@ -674,6 +676,10 @@ class AppState extends ChangeNotifier {
return await _suspectedLocationState.refreshData();
}
Future<void> reinitSuspectedLocations() async {
await _suspectedLocationState.init(offlineMode: _settingsState.offlineMode);
}
void selectSuspectedLocation(SuspectedLocation location) {
_suspectedLocationState.selectLocation(location);
}
@@ -682,13 +688,27 @@ class AppState extends ChangeNotifier {
_suspectedLocationState.clearSelection();
}
List<SuspectedLocation> getSuspectedLocationsInBounds({
Future<List<SuspectedLocation>> getSuspectedLocationsInBounds({
required double north,
required double south,
required double east,
required double west,
}) async {
return await _suspectedLocationState.getLocationsInBounds(
north: north,
south: south,
east: east,
west: west,
);
}
List<SuspectedLocation> getSuspectedLocationsInBoundsSync({
required double north,
required double south,
required double east,
required double west,
}) {
return _suspectedLocationState.getLocationsInBounds(
return _suspectedLocationState.getLocationsInBoundsSync(
north: north,
south: south,
east: east,

View File

@@ -128,6 +128,7 @@ const double kNodeProximityWarningDistance = 15.0; // meters - distance threshol
// Navigation route planning configuration
const double kNavigationMinRouteDistance = 100.0; // meters - minimum distance between start and end points
const double kNavigationDistanceWarningThreshold = 20000.0; // meters - distance threshold for timeout warning (30km)
// Node display configuration
const int kDefaultMaxNodes = 500; // Default maximum number of nodes to render on the map at once

141
lib/migrations.dart Normal file
View File

@@ -0,0 +1,141 @@
import 'dart:convert';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'app_state.dart';
import 'services/profile_service.dart';
import 'services/suspected_location_cache.dart';
import 'widgets/nuclear_reset_dialog.dart';
/// One-time migrations that run when users upgrade to specific versions.
/// Each migration function is named after the version where it should run.
class OneTimeMigrations {
/// Enable network status indicator for all existing users (v1.3.1)
static Future<void> migrate_1_3_1(AppState appState) async {
await appState.setNetworkStatusIndicatorEnabled(true);
debugPrint('[Migration] 1.3.1 completed: enabled network status indicator');
}
/// Migrate upload queue to new two-stage changeset system (v1.5.3)
static Future<void> migrate_1_5_3(AppState appState) async {
// Migration is handled automatically in PendingUpload.fromJson via _migrateFromLegacyFields
// This triggers a queue reload to apply migrations
await appState.reloadUploadQueue();
debugPrint('[Migration] 1.5.3 completed: migrated upload queue to two-stage system');
}
/// Clear FOV values from built-in profiles only (v1.6.3)
static Future<void> migrate_1_6_3(AppState appState) async {
// Load all custom profiles from storage (includes any customized built-in profiles)
final profiles = await ProfileService().load();
// Find profiles with built-in IDs and clear their FOV values
final updatedProfiles = profiles.map((profile) {
if (profile.id.startsWith('builtin-') && profile.fov != null) {
debugPrint('[Migration] Clearing FOV from profile: ${profile.id}');
return profile.copyWith(fov: null);
}
return profile;
}).toList();
// Save updated profiles back to storage
await ProfileService().save(updatedProfiles);
debugPrint('[Migration] 1.6.3 completed: cleared FOV values from built-in profiles');
}
/// Migrate suspected locations from SharedPreferences to SQLite (v1.8.0)
static Future<void> migrate_1_8_0(AppState appState) async {
try {
final prefs = await SharedPreferences.getInstance();
// Legacy SharedPreferences keys
const legacyProcessedDataKey = 'suspected_locations_processed_data';
const legacyLastFetchKey = 'suspected_locations_last_fetch';
// Check if we have legacy data
final legacyData = prefs.getString(legacyProcessedDataKey);
final legacyLastFetch = prefs.getInt(legacyLastFetchKey);
if (legacyData != null && legacyLastFetch != null) {
debugPrint('[Migration] 1.8.0: Found legacy suspected location data, migrating to database...');
// Parse legacy processed data format
final List<dynamic> legacyProcessedList = jsonDecode(legacyData);
final List<Map<String, dynamic>> rawDataList = [];
for (final entry in legacyProcessedList) {
if (entry is Map<String, dynamic> && entry['rawData'] != null) {
rawDataList.add(Map<String, dynamic>.from(entry['rawData']));
}
}
if (rawDataList.isNotEmpty) {
final fetchTime = DateTime.fromMillisecondsSinceEpoch(legacyLastFetch);
// Get the cache instance and migrate data
final cache = SuspectedLocationCache();
await cache.loadFromStorage(); // Initialize database
await cache.processAndSave(rawDataList, fetchTime);
debugPrint('[Migration] 1.8.0: Migrated ${rawDataList.length} entries from legacy storage');
}
// Clean up legacy data after successful migration
await prefs.remove(legacyProcessedDataKey);
await prefs.remove(legacyLastFetchKey);
debugPrint('[Migration] 1.8.0: Legacy data cleanup completed');
}
// Ensure suspected locations are reinitialized with new system
await appState.reinitSuspectedLocations();
debugPrint('[Migration] 1.8.0 completed: migrated suspected locations to SQLite database');
} catch (e) {
debugPrint('[Migration] 1.8.0 ERROR: Failed to migrate suspected locations: $e');
// Don't rethrow - migration failure shouldn't break the app
// The new system will work fine, users just lose their cached data
}
}
/// Get the migration function for a specific version
static Future<void> Function(AppState)? getMigrationForVersion(String version) {
switch (version) {
case '1.3.1':
return migrate_1_3_1;
case '1.5.3':
return migrate_1_5_3;
case '1.6.3':
return migrate_1_6_3;
case '1.8.0':
return migrate_1_8_0;
default:
return null;
}
}
/// Run migration for a specific version with nuclear reset on failure
static Future<void> runMigration(String version, AppState appState, BuildContext? context) async {
try {
final migration = getMigrationForVersion(version);
if (migration != null) {
await migration(appState);
} else {
debugPrint('[Migration] Unknown migration version: $version');
}
} catch (error, stackTrace) {
debugPrint('[Migration] CRITICAL: Migration $version failed: $error');
debugPrint('[Migration] Stack trace: $stackTrace');
// Nuclear option: clear everything and show non-dismissible error dialog
if (context != null) {
NuclearResetDialog.show(context, error, stackTrace);
} else {
// If no context available, just log and hope for the best
debugPrint('[Migration] No context available for error dialog, migration failure unhandled');
}
}
}
}

View File

@@ -160,7 +160,7 @@ class _HomeScreenState extends State<HomeScreen> with TickerProviderStateMixin {
// Run any needed migrations first
final versionsNeedingMigration = await ChangelogService().getVersionsNeedingMigration();
for (final version in versionsNeedingMigration) {
await ChangelogService().runMigration(version, appState);
await ChangelogService().runMigration(version, appState, context);
}
// Determine what popup to show

View File

@@ -3,9 +3,33 @@ import 'package:provider/provider.dart';
import '../../../app_state.dart';
import '../../../services/localization_service.dart';
class SuspectedLocationsSection extends StatelessWidget {
class SuspectedLocationsSection extends StatefulWidget {
const SuspectedLocationsSection({super.key});
@override
State<SuspectedLocationsSection> createState() => _SuspectedLocationsSectionState();
}
class _SuspectedLocationsSectionState extends State<SuspectedLocationsSection> {
DateTime? _lastFetch;
bool _wasLoading = false;
@override
void initState() {
super.initState();
_loadLastFetch();
}
void _loadLastFetch() async {
final appState = context.read<AppState>();
final lastFetch = await appState.suspectedLocationsLastFetch;
if (mounted) {
setState(() {
_lastFetch = lastFetch;
});
}
}
@override
Widget build(BuildContext context) {
return AnimatedBuilder(
@@ -15,14 +39,31 @@ class SuspectedLocationsSection extends StatelessWidget {
final appState = context.watch<AppState>();
final isEnabled = appState.suspectedLocationsEnabled;
final isLoading = appState.suspectedLocationsLoading;
final lastFetch = appState.suspectedLocationsLastFetch;
final downloadProgress = appState.suspectedLocationsDownloadProgress;
// Check if loading just finished and reload last fetch time
if (_wasLoading && !isLoading) {
WidgetsBinding.instance.addPostFrameCallback((_) {
_loadLastFetch();
});
}
_wasLoading = isLoading;
String getLastFetchText() {
if (lastFetch == null) {
// Show status during loading
if (isLoading) {
if (downloadProgress != null) {
return 'Downloading data... (this may take a few minutes)';
} else {
return 'Processing data...';
}
}
if (_lastFetch == null) {
return locService.t('suspectedLocations.neverFetched');
} else {
final now = DateTime.now();
final diff = now.difference(lastFetch);
final diff = now.difference(_lastFetch!);
if (diff.inDays > 0) {
return locService.t('suspectedLocations.daysAgo', params: [diff.inDays.toString()]);
} else if (diff.inHours > 0) {
@@ -42,6 +83,11 @@ class SuspectedLocationsSection extends StatelessWidget {
// The loading state will be managed by suspected location state
final success = await appState.refreshSuspectedLocations();
// Refresh the last fetch time after successful refresh
if (success) {
_loadLastFetch();
}
// Show result snackbar
if (context.mounted) {
ScaffoldMessenger.of(context).showSnackBar(
@@ -85,10 +131,31 @@ class SuspectedLocationsSection extends StatelessWidget {
title: Text(locService.t('suspectedLocations.lastUpdated')),
subtitle: Text(getLastFetchText()),
trailing: isLoading
? const SizedBox(
width: 24,
? SizedBox(
width: 80,
height: 24,
child: CircularProgressIndicator(strokeWidth: 2),
child: downloadProgress != null
? Column(
mainAxisSize: MainAxisSize.min,
children: [
LinearProgressIndicator(
value: downloadProgress,
backgroundColor: Colors.grey[300],
),
const SizedBox(height: 2),
Text(
'${(downloadProgress * 100).toInt()}%',
style: Theme.of(context).textTheme.bodySmall,
),
],
)
: const Center(
child: SizedBox(
width: 24,
height: 24,
child: CircularProgressIndicator(strokeWidth: 2),
),
),
)
: IconButton(
icon: const Icon(Icons.refresh),

View File

@@ -1,9 +1,11 @@
import 'dart:convert';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'version_service.dart';
import '../app_state.dart';
import '../migrations.dart';
/// Service for managing changelog data and first launch detection
class ChangelogService {
@@ -207,6 +209,10 @@ class ChangelogService {
versionsNeedingMigration.add('1.5.3');
}
if (needsMigration(lastSeenVersion, currentVersion, '1.6.3')) {
versionsNeedingMigration.add('1.6.3');
}
// Future versions can be added here
// if (needsMigration(lastSeenVersion, currentVersion, '2.0.0')) {
// versionsNeedingMigration.add('2.0.0');
@@ -262,31 +268,9 @@ class ChangelogService {
bool get isInitialized => _initialized;
/// Run a specific migration by version number
Future<void> runMigration(String version, AppState appState) async {
Future<void> runMigration(String version, AppState appState, BuildContext? context) async {
debugPrint('[ChangelogService] Running $version migration');
switch (version) {
case '1.3.1':
// Enable network status indicator for all existing users
await appState.setNetworkStatusIndicatorEnabled(true);
debugPrint('[ChangelogService] 1.3.1 migration completed: enabled network status indicator');
break;
case '1.5.3':
// Migrate upload queue to new two-stage changeset system
await appState.migrateUploadQueueToTwoStageSystem();
debugPrint('[ChangelogService] 1.5.3 migration completed: migrated upload queue to two-stage system');
break;
// Future migrations can be added here
// case '2.0.0':
// await appState.doSomethingNew();
// debugPrint('[ChangelogService] 2.0.0 migration completed');
// break;
default:
debugPrint('[ChangelogService] Unknown migration version: $version');
}
await OneTimeMigrations.runMigration(version, appState, context);
}
/// Check if a migration should run

View File

@@ -0,0 +1,160 @@
import 'dart:io';
import 'package:flutter/foundation.dart';
import 'package:flutter/services.dart';
import 'package:path_provider/path_provider.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'version_service.dart';
/// Nuclear reset service - clears ALL app data when migrations fail.
/// This is the "big hammer" approach for when something goes seriously wrong.
class NuclearResetService {
static final NuclearResetService _instance = NuclearResetService._();
factory NuclearResetService() => _instance;
NuclearResetService._();
/// Completely clear all app data - SharedPreferences, files, caches, everything.
/// After this, the app should behave exactly like a fresh install.
static Future<void> clearEverything() async {
try {
debugPrint('[NuclearReset] Starting complete app data wipe...');
// Clear ALL SharedPreferences
await _clearSharedPreferences();
// Clear ALL files in app directories
await _clearFileSystem();
debugPrint('[NuclearReset] Complete app data wipe finished');
} catch (e) {
// Even the nuclear option can fail, but we can't do anything about it
debugPrint('[NuclearReset] Error during nuclear reset: $e');
}
}
/// Clear all SharedPreferences data
static Future<void> _clearSharedPreferences() async {
try {
final prefs = await SharedPreferences.getInstance();
await prefs.clear();
debugPrint('[NuclearReset] Cleared SharedPreferences');
} catch (e) {
debugPrint('[NuclearReset] Failed to clear SharedPreferences: $e');
}
}
/// Clear all files and directories in app storage
static Future<void> _clearFileSystem() async {
try {
// Clear Documents directory (offline areas, etc.)
await _clearDirectory(() => getApplicationDocumentsDirectory(), 'Documents');
// Clear Cache directory (tile cache, etc.)
await _clearDirectory(() => getTemporaryDirectory(), 'Cache');
// Clear Support directory if it exists (iOS/macOS)
if (!kIsWeb && (Platform.isIOS || Platform.isMacOS)) {
await _clearDirectory(() => getApplicationSupportDirectory(), 'Support');
}
} catch (e) {
debugPrint('[NuclearReset] Failed to clear file system: $e');
}
}
/// Clear a specific directory, with error handling
static Future<void> _clearDirectory(
Future<Directory> Function() getDirFunc,
String dirName,
) async {
try {
final dir = await getDirFunc();
if (dir.existsSync()) {
await dir.delete(recursive: true);
debugPrint('[NuclearReset] Cleared $dirName directory');
}
} catch (e) {
debugPrint('[NuclearReset] Failed to clear $dirName directory: $e');
}
}
/// Generate error report information (safely, with fallbacks)
static Future<String> generateErrorReport(Object error, StackTrace? stackTrace) async {
final buffer = StringBuffer();
// Basic error information (always include this)
buffer.writeln('MIGRATION FAILURE ERROR REPORT');
buffer.writeln('Generated: ${DateTime.now().toIso8601String()}');
buffer.writeln('');
buffer.writeln('Error: $error');
if (stackTrace != null) {
buffer.writeln('');
buffer.writeln('Stack trace:');
buffer.writeln(stackTrace.toString());
}
// Try to add enrichment data, but don't fail if it doesn't work
await _addEnrichmentData(buffer);
return buffer.toString();
}
/// Add device/app information to error report (with extensive error handling)
static Future<void> _addEnrichmentData(StringBuffer buffer) async {
try {
buffer.writeln('');
buffer.writeln('--- System Information ---');
// App version (should always work)
try {
buffer.writeln('App Version: ${VersionService().version}');
} catch (e) {
buffer.writeln('App Version: [Failed to get version: $e]');
}
// Platform information
try {
if (!kIsWeb) {
buffer.writeln('Platform: ${Platform.operatingSystem}');
buffer.writeln('OS Version: ${Platform.operatingSystemVersion}');
} else {
buffer.writeln('Platform: Web');
}
} catch (e) {
buffer.writeln('Platform: [Failed to get platform info: $e]');
}
// Flutter/Dart information
try {
buffer.writeln('Flutter Mode: ${kDebugMode ? 'Debug' : kProfileMode ? 'Profile' : 'Release'}');
} catch (e) {
buffer.writeln('Flutter Mode: [Failed to get mode: $e]');
}
// Previous version (if available)
try {
final prefs = await SharedPreferences.getInstance();
final lastVersion = prefs.getString('last_seen_version');
buffer.writeln('Previous Version: ${lastVersion ?? 'Unknown (fresh install?)'}');
} catch (e) {
buffer.writeln('Previous Version: [Failed to get: $e]');
}
} catch (e) {
// If enrichment completely fails, just note it
buffer.writeln('');
buffer.writeln('--- System Information ---');
buffer.writeln('[Failed to gather system information: $e]');
}
}
/// Copy text to clipboard (safely)
static Future<void> copyToClipboard(String text) async {
try {
await Clipboard.setData(ClipboardData(text: text));
debugPrint('[NuclearReset] Copied error report to clipboard');
} catch (e) {
debugPrint('[NuclearReset] Failed to copy to clipboard: $e');
}
}
}

View File

@@ -1,130 +1,109 @@
import 'dart:convert';
import 'package:flutter/foundation.dart';
import 'package:flutter_map/flutter_map.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'package:latlong2/latlong.dart';
import '../models/suspected_location.dart';
import 'suspected_location_service.dart';
/// Lightweight entry with pre-calculated centroid for efficient bounds checking
class SuspectedLocationEntry {
final Map<String, dynamic> rawData;
final LatLng centroid;
SuspectedLocationEntry({required this.rawData, required this.centroid});
Map<String, dynamic> toJson() => {
'rawData': rawData,
'centroid': [centroid.latitude, centroid.longitude],
};
factory SuspectedLocationEntry.fromJson(Map<String, dynamic> json) {
final centroidList = json['centroid'] as List;
return SuspectedLocationEntry(
rawData: Map<String, dynamic>.from(json['rawData']),
centroid: LatLng(
(centroidList[0] as num).toDouble(),
(centroidList[1] as num).toDouble(),
),
);
}
}
import 'suspected_location_database.dart';
class SuspectedLocationCache extends ChangeNotifier {
static final SuspectedLocationCache _instance = SuspectedLocationCache._();
factory SuspectedLocationCache() => _instance;
SuspectedLocationCache._();
static const String _prefsKeyProcessedData = 'suspected_locations_processed_data';
static const String _prefsKeyLastFetch = 'suspected_locations_last_fetch';
final SuspectedLocationDatabase _database = SuspectedLocationDatabase();
List<SuspectedLocationEntry> _processedEntries = [];
DateTime? _lastFetchTime;
final Map<String, List<SuspectedLocation>> _boundsCache = {};
// Simple cache: just hold the currently visible locations
List<SuspectedLocation> _currentLocations = [];
String? _currentBoundsKey;
bool _isLoading = false;
/// Get suspected locations within specific bounds (cached)
List<SuspectedLocation> getLocationsForBounds(LatLngBounds bounds) {
/// Get suspected locations within specific bounds (async version)
Future<List<SuspectedLocation>> getLocationsForBounds(LatLngBounds bounds) async {
if (!SuspectedLocationService().isEnabled) {
debugPrint('[SuspectedLocationCache] Service not enabled');
return [];
}
final boundsKey = '${bounds.north.toStringAsFixed(4)},${bounds.south.toStringAsFixed(4)},${bounds.east.toStringAsFixed(4)},${bounds.west.toStringAsFixed(4)}';
final boundsKey = _getBoundsKey(bounds);
// debugPrint('[SuspectedLocationCache] Getting locations for bounds: $boundsKey, processed entries count: ${_processedEntries.length}');
// Check cache first
if (_boundsCache.containsKey(boundsKey)) {
debugPrint('[SuspectedLocationCache] Using cached result: ${_boundsCache[boundsKey]!.length} locations');
return _boundsCache[boundsKey]!;
// If this is the same bounds we're already showing, return current cache
if (boundsKey == _currentBoundsKey) {
return _currentLocations;
}
// Filter processed entries for this bounds (very fast since centroids are pre-calculated)
final locations = <SuspectedLocation>[];
int inBoundsCount = 0;
for (final entry in _processedEntries) {
// Quick bounds check using pre-calculated centroid
final lat = entry.centroid.latitude;
final lng = entry.centroid.longitude;
try {
// Query database for locations in bounds
final locations = await _database.getLocationsInBounds(bounds);
if (lat <= bounds.north && lat >= bounds.south &&
lng <= bounds.east && lng >= bounds.west) {
try {
// Only create SuspectedLocation object if it's in bounds
final location = SuspectedLocation.fromCsvRow(entry.rawData);
locations.add(location);
inBoundsCount++;
} catch (e) {
// Skip invalid entries
continue;
}
}
// Update cache
_currentLocations = locations;
_currentBoundsKey = boundsKey;
return locations;
} catch (e) {
debugPrint('[SuspectedLocationCache] Error querying database: $e');
return [];
}
// debugPrint('[SuspectedLocationCache] Checked ${_processedEntries.length} entries, $inBoundsCount in bounds, result: ${locations.length} locations');
// Cache the result
_boundsCache[boundsKey] = locations;
// Limit cache size to prevent memory issues
if (_boundsCache.length > 100) {
final oldestKey = _boundsCache.keys.first;
_boundsCache.remove(oldestKey);
}
return locations;
}
/// Load processed data from storage
Future<void> loadFromStorage() async {
/// Get suspected locations within specific bounds (synchronous version for UI)
/// Returns current cache immediately, triggers async update if bounds changed
List<SuspectedLocation> getLocationsForBoundsSync(LatLngBounds bounds) {
if (!SuspectedLocationService().isEnabled) {
return [];
}
final boundsKey = _getBoundsKey(bounds);
// If bounds haven't changed, return current cache immediately
if (boundsKey == _currentBoundsKey) {
return _currentLocations;
}
// Bounds changed - trigger async update but keep showing current cache
if (!_isLoading) {
_isLoading = true;
_updateCacheAsync(bounds, boundsKey);
}
// Return current cache (keeps suspected locations visible during map movement)
return _currentLocations;
}
/// Simple async update - no complex caching, just swap when done
void _updateCacheAsync(LatLngBounds bounds, String boundsKey) async {
try {
final prefs = await SharedPreferences.getInstance();
final locations = await _database.getLocationsInBounds(bounds);
// Load last fetch time
final lastFetchMs = prefs.getInt(_prefsKeyLastFetch);
if (lastFetchMs != null) {
_lastFetchTime = DateTime.fromMillisecondsSinceEpoch(lastFetchMs);
}
// Load processed data
final processedDataString = prefs.getString(_prefsKeyProcessedData);
if (processedDataString != null) {
final List<dynamic> processedDataList = jsonDecode(processedDataString);
_processedEntries = processedDataList
.map((json) => SuspectedLocationEntry.fromJson(json as Map<String, dynamic>))
.toList();
debugPrint('[SuspectedLocationCache] Loaded ${_processedEntries.length} processed entries from storage');
// Only update if this is still the most recent request
if (boundsKey == _getBoundsKey(bounds) || _currentBoundsKey == null) {
_currentLocations = locations;
_currentBoundsKey = boundsKey;
notifyListeners(); // Trigger UI update
}
} catch (e) {
debugPrint('[SuspectedLocationCache] Error loading from storage: $e');
_processedEntries.clear();
_lastFetchTime = null;
debugPrint('[SuspectedLocationCache] Error updating cache: $e');
} finally {
_isLoading = false;
}
}
/// Process raw CSV data and save to storage (calculates centroids once)
/// Generate cache key for bounds
String _getBoundsKey(LatLngBounds bounds) {
return '${bounds.north.toStringAsFixed(4)},${bounds.south.toStringAsFixed(4)},${bounds.east.toStringAsFixed(4)},${bounds.west.toStringAsFixed(4)}';
}
/// Initialize the cache (ensures database is ready)
Future<void> loadFromStorage() async {
try {
await _database.init();
debugPrint('[SuspectedLocationCache] Database initialized successfully');
} catch (e) {
debugPrint('[SuspectedLocationCache] Error initializing database: $e');
}
}
/// Process raw CSV data and save to database
Future<void> processAndSave(
List<Map<String, dynamic>> rawData,
DateTime fetchTime,
@@ -132,96 +111,39 @@ class SuspectedLocationCache extends ChangeNotifier {
try {
debugPrint('[SuspectedLocationCache] Processing ${rawData.length} raw entries...');
final processedEntries = <SuspectedLocationEntry>[];
int validCount = 0;
int errorCount = 0;
int zeroCoordCount = 0;
// Clear cache since data will change
_currentLocations = [];
_currentBoundsKey = null;
_isLoading = false;
for (int i = 0; i < rawData.length; i++) {
final rowData = rawData[i];
// Log progress every 1000 entries for debugging
if (i % 1000 == 0) {
debugPrint('[SuspectedLocationCache] Processed ${i + 1}/${rawData.length} entries...');
}
try {
// Create a temporary SuspectedLocation to extract the centroid
final tempLocation = SuspectedLocation.fromCsvRow(rowData);
// Only save if we have a valid centroid (not at 0,0)
if (tempLocation.centroid.latitude != 0 || tempLocation.centroid.longitude != 0) {
processedEntries.add(SuspectedLocationEntry(
rawData: rowData,
centroid: tempLocation.centroid,
));
validCount++;
} else {
zeroCoordCount++;
}
} catch (e) {
errorCount++;
continue;
}
}
// Insert data into database in batch
await _database.insertBatch(rawData, fetchTime);
debugPrint('[SuspectedLocationCache] Processing complete - Valid: $validCount, Zero coords: $zeroCoordCount, Errors: $errorCount');
final totalCount = await _database.getTotalCount();
debugPrint('[SuspectedLocationCache] Processed and saved $totalCount entries to database');
_processedEntries = processedEntries;
_lastFetchTime = fetchTime;
// Clear bounds cache since data changed
_boundsCache.clear();
final prefs = await SharedPreferences.getInstance();
// Save processed data
final processedDataString = jsonEncode(processedEntries.map((e) => e.toJson()).toList());
await prefs.setString(_prefsKeyProcessedData, processedDataString);
// Save last fetch time
await prefs.setInt(_prefsKeyLastFetch, fetchTime.millisecondsSinceEpoch);
// Log coordinate ranges for debugging
if (processedEntries.isNotEmpty) {
double minLat = processedEntries.first.centroid.latitude;
double maxLat = minLat;
double minLng = processedEntries.first.centroid.longitude;
double maxLng = minLng;
for (final entry in processedEntries) {
final lat = entry.centroid.latitude;
final lng = entry.centroid.longitude;
if (lat < minLat) minLat = lat;
if (lat > maxLat) maxLat = lat;
if (lng < minLng) minLng = lng;
if (lng > maxLng) maxLng = lng;
}
debugPrint('[SuspectedLocationCache] Coordinate ranges - Lat: $minLat to $maxLat, Lng: $minLng to $maxLng');
}
debugPrint('[SuspectedLocationCache] Processed and saved $validCount valid entries (${processedEntries.length} total)');
notifyListeners();
} catch (e) {
debugPrint('[SuspectedLocationCache] Error processing and saving: $e');
rethrow;
}
}
/// Clear all cached data
void clear() {
_processedEntries.clear();
_boundsCache.clear();
_lastFetchTime = null;
Future<void> clear() async {
_currentLocations = [];
_currentBoundsKey = null;
_isLoading = false;
await _database.clearAllData();
notifyListeners();
}
/// Get last fetch time
DateTime? get lastFetchTime => _lastFetchTime;
Future<DateTime?> get lastFetchTime => _database.getLastFetchTime();
/// Get total count of processed entries
int get totalCount => _processedEntries.length;
Future<int> get totalCount => _database.getTotalCount();
/// Check if we have data
bool get hasData => _processedEntries.isNotEmpty;
Future<bool> get hasData => _database.hasData();
}

View File

@@ -0,0 +1,330 @@
import 'dart:convert';
import 'package:flutter/foundation.dart';
import 'package:flutter_map/flutter_map.dart';
import 'package:latlong2/latlong.dart';
import 'package:sqflite/sqflite.dart';
import 'package:path/path.dart' as path;
import '../models/suspected_location.dart';
/// Database service for suspected location data
/// Replaces the SharedPreferences-based cache to handle large datasets efficiently
class SuspectedLocationDatabase {
static final SuspectedLocationDatabase _instance = SuspectedLocationDatabase._();
factory SuspectedLocationDatabase() => _instance;
SuspectedLocationDatabase._();
Database? _database;
static const String _dbName = 'suspected_locations.db';
static const int _dbVersion = 1;
// Table and column names
static const String _tableName = 'suspected_locations';
static const String _columnTicketNo = 'ticket_no';
static const String _columnCentroidLat = 'centroid_lat';
static const String _columnCentroidLng = 'centroid_lng';
static const String _columnBounds = 'bounds';
static const String _columnGeoJson = 'geo_json';
static const String _columnAllFields = 'all_fields';
// Metadata table for tracking last fetch time
static const String _metaTableName = 'metadata';
static const String _metaColumnKey = 'key';
static const String _metaColumnValue = 'value';
static const String _lastFetchKey = 'last_fetch_time';
/// Initialize the database
Future<void> init() async {
if (_database != null) return;
try {
final dbPath = await getDatabasesPath();
final fullPath = path.join(dbPath, _dbName);
debugPrint('[SuspectedLocationDatabase] Initializing database at $fullPath');
_database = await openDatabase(
fullPath,
version: _dbVersion,
onCreate: _createTables,
onUpgrade: _upgradeTables,
);
debugPrint('[SuspectedLocationDatabase] Database initialized successfully');
} catch (e) {
debugPrint('[SuspectedLocationDatabase] Error initializing database: $e');
rethrow;
}
}
/// Create database tables
Future<void> _createTables(Database db, int version) async {
debugPrint('[SuspectedLocationDatabase] Creating tables...');
// Main suspected locations table
await db.execute('''
CREATE TABLE $_tableName (
$_columnTicketNo TEXT PRIMARY KEY,
$_columnCentroidLat REAL NOT NULL,
$_columnCentroidLng REAL NOT NULL,
$_columnBounds TEXT,
$_columnGeoJson TEXT,
$_columnAllFields TEXT NOT NULL
)
''');
// Create spatial indexes for efficient bounds queries
// Separate indexes for lat and lng for better query optimization
await db.execute('''
CREATE INDEX idx_lat ON $_tableName ($_columnCentroidLat)
''');
await db.execute('''
CREATE INDEX idx_lng ON $_tableName ($_columnCentroidLng)
''');
// Composite index for combined lat/lng queries
await db.execute('''
CREATE INDEX idx_lat_lng ON $_tableName ($_columnCentroidLat, $_columnCentroidLng)
''');
// Metadata table for tracking last fetch time and other info
await db.execute('''
CREATE TABLE $_metaTableName (
$_metaColumnKey TEXT PRIMARY KEY,
$_metaColumnValue TEXT NOT NULL
)
''');
debugPrint('[SuspectedLocationDatabase] Tables created successfully');
}
/// Handle database upgrades
Future<void> _upgradeTables(Database db, int oldVersion, int newVersion) async {
debugPrint('[SuspectedLocationDatabase] Upgrading database from version $oldVersion to $newVersion');
// Future migrations would go here
}
/// Get database instance, initializing if needed
Future<Database> get database async {
if (_database == null) {
await init();
}
return _database!;
}
/// Clear all data and recreate tables
Future<void> clearAllData() async {
try {
final db = await database;
debugPrint('[SuspectedLocationDatabase] Clearing all data...');
// Drop and recreate tables (simpler than DELETE for large datasets)
// Indexes are automatically dropped with tables
await db.execute('DROP TABLE IF EXISTS $_tableName');
await db.execute('DROP TABLE IF EXISTS $_metaTableName');
await _createTables(db, _dbVersion);
debugPrint('[SuspectedLocationDatabase] All data cleared successfully');
} catch (e) {
debugPrint('[SuspectedLocationDatabase] Error clearing data: $e');
rethrow;
}
}
/// Insert suspected locations in batch
Future<void> insertBatch(List<Map<String, dynamic>> rawDataList, DateTime fetchTime) async {
try {
final db = await database;
debugPrint('[SuspectedLocationDatabase] Starting batch insert of ${rawDataList.length} entries...');
// Clear existing data first
await clearAllData();
// Process entries in batches to avoid memory issues
const batchSize = 1000;
int totalInserted = 0;
int validCount = 0;
int errorCount = 0;
// Start transaction for better performance
await db.transaction((txn) async {
for (int i = 0; i < rawDataList.length; i += batchSize) {
final batch = txn.batch();
final endIndex = (i + batchSize < rawDataList.length) ? i + batchSize : rawDataList.length;
final currentBatch = rawDataList.sublist(i, endIndex);
for (final rowData in currentBatch) {
try {
// Create temporary SuspectedLocation to extract centroid and bounds
final tempLocation = SuspectedLocation.fromCsvRow(rowData);
// Skip entries with zero coordinates
if (tempLocation.centroid.latitude == 0 && tempLocation.centroid.longitude == 0) {
continue;
}
// Prepare data for database insertion
final dbRow = {
_columnTicketNo: tempLocation.ticketNo,
_columnCentroidLat: tempLocation.centroid.latitude,
_columnCentroidLng: tempLocation.centroid.longitude,
_columnBounds: tempLocation.bounds.isNotEmpty
? jsonEncode(tempLocation.bounds.map((p) => [p.latitude, p.longitude]).toList())
: null,
_columnGeoJson: tempLocation.geoJson != null ? jsonEncode(tempLocation.geoJson!) : null,
_columnAllFields: jsonEncode(tempLocation.allFields),
};
batch.insert(_tableName, dbRow, conflictAlgorithm: ConflictAlgorithm.replace);
validCount++;
} catch (e) {
errorCount++;
// Skip invalid entries
continue;
}
}
// Commit this batch
await batch.commit(noResult: true);
totalInserted += currentBatch.length;
// Log progress every few batches
if ((i ~/ batchSize) % 5 == 0) {
debugPrint('[SuspectedLocationDatabase] Processed ${i + currentBatch.length}/${rawDataList.length} entries...');
}
}
// Insert metadata
await txn.insert(
_metaTableName,
{
_metaColumnKey: _lastFetchKey,
_metaColumnValue: fetchTime.millisecondsSinceEpoch.toString(),
},
conflictAlgorithm: ConflictAlgorithm.replace,
);
});
debugPrint('[SuspectedLocationDatabase] Batch insert complete - Valid: $validCount, Errors: $errorCount');
} catch (e) {
debugPrint('[SuspectedLocationDatabase] Error in batch insert: $e');
rethrow;
}
}
/// Get suspected locations within bounding box
Future<List<SuspectedLocation>> getLocationsInBounds(LatLngBounds bounds) async {
try {
final db = await database;
// Query with spatial bounds (simple lat/lng box filtering)
final result = await db.query(
_tableName,
where: '''
$_columnCentroidLat <= ? AND $_columnCentroidLat >= ? AND
$_columnCentroidLng <= ? AND $_columnCentroidLng >= ?
''',
whereArgs: [bounds.north, bounds.south, bounds.east, bounds.west],
);
// Convert database rows to SuspectedLocation objects
final locations = <SuspectedLocation>[];
for (final row in result) {
try {
final allFields = Map<String, dynamic>.from(jsonDecode(row[_columnAllFields] as String));
// Reconstruct bounds if available
List<LatLng> boundsList = [];
final boundsJson = row[_columnBounds] as String?;
if (boundsJson != null) {
final boundsData = jsonDecode(boundsJson) as List;
boundsList = boundsData.map((b) => LatLng(
(b[0] as num).toDouble(),
(b[1] as num).toDouble(),
)).toList();
}
// Reconstruct GeoJSON if available
Map<String, dynamic>? geoJson;
final geoJsonString = row[_columnGeoJson] as String?;
if (geoJsonString != null) {
geoJson = Map<String, dynamic>.from(jsonDecode(geoJsonString));
}
final location = SuspectedLocation(
ticketNo: row[_columnTicketNo] as String,
centroid: LatLng(
row[_columnCentroidLat] as double,
row[_columnCentroidLng] as double,
),
bounds: boundsList,
geoJson: geoJson,
allFields: allFields,
);
locations.add(location);
} catch (e) {
// Skip invalid database entries
debugPrint('[SuspectedLocationDatabase] Error parsing row: $e');
continue;
}
}
return locations;
} catch (e) {
debugPrint('[SuspectedLocationDatabase] Error querying bounds: $e');
return [];
}
}
/// Get last fetch time
Future<DateTime?> getLastFetchTime() async {
try {
final db = await database;
final result = await db.query(
_metaTableName,
where: '$_metaColumnKey = ?',
whereArgs: [_lastFetchKey],
);
if (result.isNotEmpty) {
final value = result.first[_metaColumnValue] as String;
return DateTime.fromMillisecondsSinceEpoch(int.parse(value));
}
return null;
} catch (e) {
debugPrint('[SuspectedLocationDatabase] Error getting last fetch time: $e');
return null;
}
}
/// Get total count of entries
Future<int> getTotalCount() async {
try {
final db = await database;
final result = await db.rawQuery('SELECT COUNT(*) as count FROM $_tableName');
return Sqflite.firstIntValue(result) ?? 0;
} catch (e) {
debugPrint('[SuspectedLocationDatabase] Error getting total count: $e');
return 0;
}
}
/// Check if database has data
Future<bool> hasData() async {
final count = await getTotalCount();
return count > 0;
}
/// Close database connection
Future<void> close() async {
if (_database != null) {
await _database!.close();
_database = null;
}
}
}

View File

@@ -18,13 +18,13 @@ class SuspectedLocationService {
static const String _prefsKeyEnabled = 'suspected_locations_enabled';
static const Duration _maxAge = Duration(days: 7);
static const Duration _timeout = Duration(seconds: 30);
static const Duration _timeout = Duration(minutes: 5); // Increased for large CSV files (100MB+)
final SuspectedLocationCache _cache = SuspectedLocationCache();
bool _isEnabled = false;
/// Get last fetch time
DateTime? get lastFetchTime => _cache.lastFetchTime;
Future<DateTime?> get lastFetchTime => _cache.lastFetchTime;
/// Check if suspected locations are enabled
bool get isEnabled => _isEnabled;
@@ -37,11 +37,12 @@ class SuspectedLocationService {
await _cache.loadFromStorage();
// Only auto-fetch if enabled, data is stale or missing, and we are not offline
if (_isEnabled && _shouldRefresh() && !offlineMode) {
if (_isEnabled && (await _shouldRefresh()) && !offlineMode) {
debugPrint('[SuspectedLocationService] Auto-refreshing CSV data on startup (older than $_maxAge or missing)');
await _fetchData();
} else if (_isEnabled && _shouldRefresh() && offlineMode) {
debugPrint('[SuspectedLocationService] Skipping auto-refresh due to offline mode - data is ${_cache.lastFetchTime != null ? 'outdated' : 'missing'}');
} else if (_isEnabled && (await _shouldRefresh()) && offlineMode) {
final lastFetch = await _cache.lastFetchTime;
debugPrint('[SuspectedLocationService] Skipping auto-refresh due to offline mode - data is ${lastFetch != null ? 'outdated' : 'missing'}');
}
}
@@ -53,36 +54,37 @@ class SuspectedLocationService {
// If disabling, clear the cache
if (!enabled) {
_cache.clear();
await _cache.clear();
}
// Note: If enabling and no data, the state layer will call fetchDataIfNeeded()
}
/// Check if cache has any data
bool get hasData => _cache.hasData;
Future<bool> get hasData => _cache.hasData;
/// Get last fetch time
DateTime? get lastFetch => _cache.lastFetchTime;
Future<DateTime?> get lastFetch => _cache.lastFetchTime;
/// Fetch data if needed (for enabling suspected locations when no data exists)
Future<bool> fetchDataIfNeeded() async {
if (!_shouldRefresh()) {
Future<bool> fetchDataIfNeeded({void Function(double)? onProgress}) async {
if (!(await _shouldRefresh())) {
debugPrint('[SuspectedLocationService] Data is fresh, skipping fetch');
return true; // Already have fresh data
}
return await _fetchData();
return await _fetchData(onProgress: onProgress);
}
/// Force refresh the data (for manual refresh button)
Future<bool> forceRefresh() async {
return await _fetchData();
Future<bool> forceRefresh({void Function(double)? onProgress}) async {
return await _fetchData(onProgress: onProgress);
}
/// Check if data should be refreshed
bool _shouldRefresh() {
if (!_cache.hasData) return true;
if (_cache.lastFetchTime == null) return true;
return DateTime.now().difference(_cache.lastFetchTime!) > _maxAge;
Future<bool> _shouldRefresh() async {
if (!(await _cache.hasData)) return true;
final lastFetch = await _cache.lastFetchTime;
if (lastFetch == null) return true;
return DateTime.now().difference(lastFetch) > _maxAge;
}
/// Load settings from shared preferences
@@ -100,111 +102,175 @@ class SuspectedLocationService {
}
/// Fetch data from the CSV URL
Future<bool> _fetchData() async {
try {
debugPrint('[SuspectedLocationService] Fetching CSV data from $kSuspectedLocationsCsvUrl');
final response = await http.get(
Uri.parse(kSuspectedLocationsCsvUrl),
headers: {
'User-Agent': 'DeFlock/1.0 (OSM surveillance mapping app)',
},
).timeout(_timeout);
if (response.statusCode != 200) {
debugPrint('[SuspectedLocationService] HTTP error ${response.statusCode}');
return false;
}
// Parse CSV with proper field separator and quote handling
final csvData = await compute(_parseCSV, response.body);
debugPrint('[SuspectedLocationService] Parsed ${csvData.length} rows from CSV');
if (csvData.isEmpty) {
debugPrint('[SuspectedLocationService] Empty CSV data');
return false;
}
// First row should be headers
final headers = csvData.first.map((h) => h.toString().toLowerCase()).toList();
debugPrint('[SuspectedLocationService] Headers: $headers');
final dataRows = csvData.skip(1);
debugPrint('[SuspectedLocationService] Data rows count: ${dataRows.length}');
// Find required column indices - we only need ticket_no and location
final ticketNoIndex = headers.indexOf('ticket_no');
final locationIndex = headers.indexOf('location');
debugPrint('[SuspectedLocationService] Column indices - ticket_no: $ticketNoIndex, location: $locationIndex');
if (ticketNoIndex == -1 || locationIndex == -1) {
debugPrint('[SuspectedLocationService] Required columns not found in CSV. Headers: $headers');
return false;
}
// Parse rows and store all data dynamically
final List<Map<String, dynamic>> rawDataList = [];
int rowIndex = 0;
int validRows = 0;
for (final row in dataRows) {
rowIndex++;
try {
final Map<String, dynamic> rowData = {};
Future<bool> _fetchData({void Function(double)? onProgress}) async {
const maxRetries = 3;
for (int attempt = 1; attempt <= maxRetries; attempt++) {
try {
debugPrint('[SuspectedLocationService] Fetching CSV data from $kSuspectedLocationsCsvUrl (attempt $attempt/$maxRetries)');
if (attempt == 1) {
debugPrint('[SuspectedLocationService] This may take up to ${_timeout.inMinutes} minutes for large datasets...');
}
// Use streaming download for progress tracking
final request = http.Request('GET', Uri.parse(kSuspectedLocationsCsvUrl));
request.headers['User-Agent'] = 'DeFlock/1.0 (OSM surveillance mapping app)';
final client = http.Client();
final streamedResponse = await client.send(request).timeout(_timeout);
if (streamedResponse.statusCode != 200) {
debugPrint('[SuspectedLocationService] HTTP error ${streamedResponse.statusCode}');
client.close();
throw Exception('HTTP ${streamedResponse.statusCode}');
}
final contentLength = streamedResponse.contentLength;
debugPrint('[SuspectedLocationService] Starting download of ${contentLength != null ? '$contentLength bytes' : 'unknown size'}...');
// Download with progress tracking
final chunks = <List<int>>[];
int downloadedBytes = 0;
await for (final chunk in streamedResponse.stream) {
chunks.add(chunk);
downloadedBytes += chunk.length;
// Store all columns dynamically
for (int i = 0; i < headers.length && i < row.length; i++) {
final headerName = headers[i];
final cellValue = row[i];
if (cellValue != null) {
rowData[headerName] = cellValue;
// Report progress if we know the total size
if (contentLength != null && onProgress != null) {
try {
final progress = downloadedBytes / contentLength;
onProgress(progress.clamp(0.0, 1.0));
} catch (e) {
// Don't let progress callback errors break the download
debugPrint('[SuspectedLocationService] Progress callback error: $e');
}
}
// Basic validation - must have ticket_no and location
if (rowData['ticket_no']?.toString().isNotEmpty == true &&
rowData['location']?.toString().isNotEmpty == true) {
rawDataList.add(rowData);
validRows++;
}
client.close();
// Combine chunks into single response body
final bodyBytes = chunks.expand((chunk) => chunk).toList();
final responseBody = String.fromCharCodes(bodyBytes);
debugPrint('[SuspectedLocationService] Downloaded $downloadedBytes bytes, parsing CSV...');
// Parse CSV with proper field separator and quote handling
final csvData = await compute(_parseCSV, responseBody);
debugPrint('[SuspectedLocationService] Parsed ${csvData.length} rows from CSV');
if (csvData.isEmpty) {
debugPrint('[SuspectedLocationService] Empty CSV data');
throw Exception('Empty CSV data');
}
// First row should be headers
final headers = csvData.first.map((h) => h.toString().toLowerCase()).toList();
debugPrint('[SuspectedLocationService] Headers: $headers');
final dataRows = csvData.skip(1);
debugPrint('[SuspectedLocationService] Data rows count: ${dataRows.length}');
// Find required column indices - we only need ticket_no and location
final ticketNoIndex = headers.indexOf('ticket_no');
final locationIndex = headers.indexOf('location');
debugPrint('[SuspectedLocationService] Column indices - ticket_no: $ticketNoIndex, location: $locationIndex');
if (ticketNoIndex == -1 || locationIndex == -1) {
debugPrint('[SuspectedLocationService] Required columns not found in CSV. Headers: $headers');
throw Exception('Required columns not found in CSV');
}
// Parse rows and store all data dynamically
final List<Map<String, dynamic>> rawDataList = [];
int rowIndex = 0;
int validRows = 0;
for (final row in dataRows) {
rowIndex++;
try {
final Map<String, dynamic> rowData = {};
// Store all columns dynamically
for (int i = 0; i < headers.length && i < row.length; i++) {
final headerName = headers[i];
final cellValue = row[i];
if (cellValue != null) {
rowData[headerName] = cellValue;
}
}
// Basic validation - must have ticket_no and location
if (rowData['ticket_no']?.toString().isNotEmpty == true &&
rowData['location']?.toString().isNotEmpty == true) {
rawDataList.add(rowData);
validRows++;
}
} catch (e, stackTrace) {
// Skip rows that can't be parsed
debugPrint('[SuspectedLocationService] Error parsing row $rowIndex: $e');
continue;
}
} catch (e, stackTrace) {
// Skip rows that can't be parsed
debugPrint('[SuspectedLocationService] Error parsing row $rowIndex: $e');
continue;
}
debugPrint('[SuspectedLocationService] Parsed $validRows valid rows from ${dataRows.length} total rows');
final fetchTime = DateTime.now();
// Process raw data and save (calculates centroids once)
await _cache.processAndSave(rawDataList, fetchTime);
debugPrint('[SuspectedLocationService] Successfully fetched and stored $validRows valid raw entries (${rawDataList.length} total)');
return true;
} catch (e, stackTrace) {
debugPrint('[SuspectedLocationService] Attempt $attempt failed: $e');
if (attempt == maxRetries) {
debugPrint('[SuspectedLocationService] All $maxRetries attempts failed');
debugPrint('[SuspectedLocationService] Stack trace: $stackTrace');
return false;
} else {
// Wait before retrying (exponential backoff)
final delay = Duration(seconds: attempt * 10);
debugPrint('[SuspectedLocationService] Retrying in ${delay.inSeconds} seconds...');
await Future.delayed(delay);
}
}
debugPrint('[SuspectedLocationService] Parsed $validRows valid rows from ${dataRows.length} total rows');
final fetchTime = DateTime.now();
// Process raw data and save (calculates centroids once)
await _cache.processAndSave(rawDataList, fetchTime);
debugPrint('[SuspectedLocationService] Successfully fetched and stored $validRows valid raw entries (${rawDataList.length} total)');
return true;
} catch (e, stackTrace) {
debugPrint('[SuspectedLocationService] Error fetching data: $e');
debugPrint('[SuspectedLocationService] Stack trace: $stackTrace');
return false;
}
return false; // Should never reach here
}
/// Get suspected locations within a bounding box
List<SuspectedLocation> getLocationsInBounds({
/// Get suspected locations within a bounding box (async)
Future<List<SuspectedLocation>> getLocationsInBounds({
required double north,
required double south,
required double east,
required double west,
}) async {
return await _cache.getLocationsForBounds(LatLngBounds(
LatLng(north, west),
LatLng(south, east),
));
}
/// Get suspected locations within a bounding box (sync, for UI)
List<SuspectedLocation> getLocationsInBoundsSync({
required double north,
required double south,
required double east,
required double west,
}) {
return _cache.getLocationsForBounds(LatLngBounds(
return _cache.getLocationsForBoundsSync(LatLngBounds(
LatLng(north, west),
LatLng(south, east),
));
}
}
/// Simple CSV parser for compute() - must be top-level function
List<List<dynamic>> _parseCSV(String csvBody) {
return const CsvToListConverter(

View File

@@ -87,6 +87,24 @@ class NavigationState extends ChangeNotifier {
return distance < kNavigationMinRouteDistance;
}
/// Get distance from first navigation point to provisional location during second point selection
double? get distanceFromFirstPoint {
if (!_isSettingSecondPoint || _provisionalPinLocation == null) return null;
final firstPoint = _nextPointIsStart ? _routeEnd : _routeStart;
if (firstPoint == null) return null;
return const Distance().as(LengthUnit.Meter, firstPoint, _provisionalPinLocation!);
}
/// Check if distance between points would likely cause timeout issues
bool get distanceExceedsWarningThreshold {
final distance = distanceFromFirstPoint;
if (distance == null) return false;
return distance > kNavigationDistanceWarningThreshold;
}
/// BRUTALIST: Single entry point to search mode
void enterSearchMode(LatLng mapCenter) {
debugPrint('[NavigationState] enterSearchMode - current mode: $_mode');

View File

@@ -8,18 +8,34 @@ class SuspectedLocationState extends ChangeNotifier {
SuspectedLocation? _selectedLocation;
bool _isLoading = false;
double? _downloadProgress; // 0.0 to 1.0, null when not downloading
/// Currently selected suspected location (for detail view)
SuspectedLocation? get selectedLocation => _selectedLocation;
/// Get suspected locations in bounds (this should be called by the map view)
List<SuspectedLocation> getLocationsInBounds({
/// Get suspected locations in bounds (async)
Future<List<SuspectedLocation>> getLocationsInBounds({
required double north,
required double south,
required double east,
required double west,
}) async {
return await _service.getLocationsInBounds(
north: north,
south: south,
east: east,
west: west,
);
}
/// Get suspected locations in bounds (sync, for UI)
List<SuspectedLocation> getLocationsInBoundsSync({
required double north,
required double south,
required double east,
required double west,
}) {
return _service.getLocationsInBounds(
return _service.getLocationsInBoundsSync(
north: north,
south: south,
east: east,
@@ -32,9 +48,12 @@ class SuspectedLocationState extends ChangeNotifier {
/// Whether currently loading data
bool get isLoading => _isLoading;
/// Download progress (0.0 to 1.0), null when not downloading
double? get downloadProgress => _downloadProgress;
/// Last time data was fetched
DateTime? get lastFetchTime => _service.lastFetchTime;
Future<DateTime?> get lastFetchTime => _service.lastFetchTime;
/// Initialize the state
Future<void> init({bool offlineMode = false}) async {
@@ -47,7 +66,7 @@ class SuspectedLocationState extends ChangeNotifier {
await _service.setEnabled(enabled);
// If enabling and no data exists, fetch it now
if (enabled && !_service.hasData) {
if (enabled && !(await _service.hasData)) {
await _fetchData();
}
@@ -57,13 +76,15 @@ class SuspectedLocationState extends ChangeNotifier {
/// Manually refresh the data (force refresh)
Future<bool> refreshData() async {
_isLoading = true;
_downloadProgress = null;
notifyListeners();
try {
final success = await _service.forceRefresh();
final success = await _service.forceRefresh(onProgress: _updateDownloadProgress);
return success;
} finally {
_isLoading = false;
_downloadProgress = null;
notifyListeners();
}
}
@@ -71,16 +92,24 @@ class SuspectedLocationState extends ChangeNotifier {
/// Internal method to fetch data if needed with loading state management
Future<bool> _fetchData() async {
_isLoading = true;
_downloadProgress = null;
notifyListeners();
try {
final success = await _service.fetchDataIfNeeded();
final success = await _service.fetchDataIfNeeded(onProgress: _updateDownloadProgress);
return success;
} finally {
_isLoading = false;
_downloadProgress = null;
notifyListeners();
}
}
/// Update download progress
void _updateDownloadProgress(double progress) {
_downloadProgress = progress;
notifyListeners();
}
/// Select a suspected location for detail view
void selectLocation(SuspectedLocation location) {

View File

@@ -84,7 +84,7 @@ class MarkerLayerBuilder {
final suspectedLocationMarkers = <Marker>[];
if (appState.suspectedLocationsEnabled && mapBounds != null &&
currentZoom >= (appState.uploadMode == UploadMode.sandbox ? kOsmApiMinZoomLevel : kNodeMinZoomLevel)) {
final suspectedLocations = appState.getSuspectedLocationsInBounds(
final suspectedLocations = appState.getSuspectedLocationsInBoundsSync(
north: mapBounds.north,
south: mapBounds.south,
east: mapBounds.east,

View File

@@ -161,7 +161,50 @@ class NavigationSheet extends StatelessWidget {
coordinates: provisionalLocation,
address: provisionalAddress,
),
const SizedBox(height: 16),
const SizedBox(height: 8),
// Show distance from first point
if (appState.distanceFromFirstPoint != null) ...[
Text(
'Distance: ${(appState.distanceFromFirstPoint! / 1000).toStringAsFixed(1)} km',
style: TextStyle(
fontSize: 14,
fontWeight: FontWeight.w500,
color: Colors.grey[600],
),
),
const SizedBox(height: 8),
],
// Show distance warning if threshold exceeded
if (appState.distanceExceedsWarningThreshold) ...[
Container(
padding: const EdgeInsets.all(12),
decoration: BoxDecoration(
color: Colors.amber.withOpacity(0.1),
borderRadius: BorderRadius.circular(8),
border: Border.all(color: Colors.amber.withOpacity(0.3)),
),
child: Row(
crossAxisAlignment: CrossAxisAlignment.start,
children: [
Icon(Icons.warning_amber, color: Colors.amber[700], size: 20),
const SizedBox(width: 8),
Expanded(
child: Text(
'Trips longer than ${(kNavigationDistanceWarningThreshold / 1000).toStringAsFixed(0)} km are likely to time out. We are working to improve this; stay tuned.',
style: TextStyle(
fontSize: 14,
color: Colors.amber[700],
),
),
),
],
),
),
const SizedBox(height: 8),
],
// Show warning message if locations are too close
if (appState.areRoutePointsTooClose) ...[

View File

@@ -0,0 +1,107 @@
import 'package:flutter/material.dart';
import 'package:url_launcher/url_launcher.dart';
import '../services/nuclear_reset_service.dart';
/// Non-dismissible error dialog shown when migrations fail and nuclear reset is triggered.
/// Forces user to restart the app by making it impossible to close this dialog.
class NuclearResetDialog extends StatelessWidget {
final String errorReport;
const NuclearResetDialog({
Key? key,
required this.errorReport,
}) : super(key: key);
@override
Widget build(BuildContext context) {
return WillPopScope(
// Prevent back button from closing dialog
onWillPop: () async => false,
child: AlertDialog(
title: const Row(
children: [
Icon(Icons.warning, color: Colors.red),
SizedBox(width: 8),
Text('Migration Error'),
],
),
content: const Column(
mainAxisSize: MainAxisSize.min,
crossAxisAlignment: CrossAxisAlignment.start,
children: [
Text(
'Unfortunately we encountered an issue during the app update and had to clear your settings and data.',
style: TextStyle(fontWeight: FontWeight.w500),
),
SizedBox(height: 12),
Text(
'You will need to:',
style: TextStyle(fontWeight: FontWeight.w500),
),
SizedBox(height: 4),
Text('• Log back into OpenStreetMap'),
Text('• Recreate any custom profiles'),
Text('• Re-download any offline areas'),
SizedBox(height: 12),
Text(
'Please close and restart the app to continue.',
style: TextStyle(fontWeight: FontWeight.w500),
),
],
),
actions: [
TextButton.icon(
onPressed: () => _copyErrorToClipboard(),
icon: const Icon(Icons.copy),
label: const Text('Copy Error'),
),
TextButton.icon(
onPressed: () => _sendErrorToSupport(),
icon: const Icon(Icons.email),
label: const Text('Send to Support'),
),
],
// No dismiss button - forces user to restart app
),
);
}
Future<void> _copyErrorToClipboard() async {
await NuclearResetService.copyToClipboard(errorReport);
}
Future<void> _sendErrorToSupport() async {
const supportEmail = 'app@deflock.me';
const subject = 'DeFlock App Migration Error Report';
// Create mailto URL with pre-filled error report
final body = Uri.encodeComponent(errorReport);
final mailtoUrl = 'mailto:$supportEmail?subject=${Uri.encodeComponent(subject)}&body=$body';
try {
final uri = Uri.parse(mailtoUrl);
if (await canLaunchUrl(uri)) {
await launchUrl(uri);
}
} catch (e) {
// If email fails, just copy to clipboard as fallback
await _copyErrorToClipboard();
}
}
/// Show the nuclear reset dialog (non-dismissible)
static Future<void> show(BuildContext context, Object error, StackTrace? stackTrace) async {
// Generate error report
final errorReport = await NuclearResetService.generateErrorReport(error, stackTrace);
// Clear all app data
await NuclearResetService.clearEverything();
// Show non-dismissible dialog
await showDialog(
context: context,
barrierDismissible: false, // Prevent tap-outside to dismiss
builder: (context) => NuclearResetDialog(errorReport: errorReport),
);
}
}

View File

@@ -484,7 +484,7 @@ packages:
source: hosted
version: "3.2.1"
path:
dependency: transitive
dependency: "direct main"
description:
name: path
sha256: "75cca69d1490965be98c73ceaea117e8a04dd21217b37b292c9ddbec0d955bc5"
@@ -680,6 +680,46 @@ packages:
url: "https://pub.dev"
source: hosted
version: "7.0.0"
sqflite:
dependency: "direct main"
description:
name: sqflite
sha256: e2297b1da52f127bc7a3da11439985d9b536f75070f3325e62ada69a5c585d03
url: "https://pub.dev"
source: hosted
version: "2.4.2"
sqflite_android:
dependency: transitive
description:
name: sqflite_android
sha256: "2b3070c5fa881839f8b402ee4a39c1b4d561704d4ebbbcfb808a119bc2a1701b"
url: "https://pub.dev"
source: hosted
version: "2.4.1"
sqflite_common:
dependency: transitive
description:
name: sqflite_common
sha256: "6ef422a4525ecc601db6c0a2233ff448c731307906e92cabc9ba292afaae16a6"
url: "https://pub.dev"
source: hosted
version: "2.5.6"
sqflite_darwin:
dependency: transitive
description:
name: sqflite_darwin
sha256: "279832e5cde3fe99e8571879498c9211f3ca6391b0d818df4e17d9fff5c6ccb3"
url: "https://pub.dev"
source: hosted
version: "2.4.2"
sqflite_platform_interface:
dependency: transitive
description:
name: sqflite_platform_interface
sha256: "8dd4515c7bdcae0a785b0062859336de775e8c65db81ae33dd5445f35be61920"
url: "https://pub.dev"
source: hosted
version: "2.4.0"
string_scanner:
dependency: transitive
description:
@@ -688,6 +728,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "1.4.1"
synchronized:
dependency: transitive
description:
name: synchronized
sha256: c254ade258ec8282947a0acbbc90b9575b4f19673533ee46f2f6e9b3aeefd7c0
url: "https://pub.dev"
source: hosted
version: "3.4.0"
term_glyph:
dependency: transitive
description:

View File

@@ -1,7 +1,7 @@
name: deflockapp
description: Map public surveillance infrastructure with OpenStreetMap
publish_to: "none"
version: 1.6.3+29 # The thing after the + is the version code, incremented with each release
version: 1.8.1+31 # The thing after the + is the version code, incremented with each release
environment:
sdk: ">=3.5.0 <4.0.0" # oauth2_client 4.x needs Dart 3.5+
@@ -30,6 +30,8 @@ dependencies:
# Persistence
shared_preferences: ^2.2.2
sqflite: ^2.4.1
path: ^1.8.3
uuid: ^4.0.0
package_info_plus: ^8.0.0
csv: ^6.0.0