How It Works

Understanding the web scraping architecture and data flow in Mi API BCV

Architecture Overview

Mi API BCV uses a combination of web scraping and API integration to collect exchange rates from multiple Venezuelan banks. The system is built on Laravel 12 and follows clean architecture principles using the Strategy pattern.

Core Components

1. ScraperService

The ScraperService is the heart of the scraping system. It handles HTTP requests and HTML parsing.

app/Services/ScraperService.php

public function scrapeData(string $url, string $banco)
{
    $response = Http::withoutVerifying()->withHeaders([
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9',
        'Accept-Language' => 'es-ES,es;q=0.9,en;q=0.8',
        'Referer' => $url,
    ])->get($url);

    if ($response->status() !== 200) {
        return null;
    }

    $crawler = new Crawler($response->body());

    return match ($banco) {
        'banplus' => $this->parseBanplusData($crawler),
        'bnc' => $this->parseBNCData($crawler),
        'bcv' => $this->parseBCVData($crawler),
        default  => 0.00,
    };
}

Key features:

• Uses realistic browser headers to avoid blocking
• Leverages Symfony DomCrawler for HTML parsing
• Bank-specific parsing methods for different website structures
• Returns null on failure, allowing the system to continue

2. Strategy Pattern

Each bank has its own strategy class implementing the UrlStrategy interface:

app/Interfaces/UrlStrategy.php

interface UrlStrategy
{
    public function getValue(): float;
}

This allows easy addition of new banks without modifying existing code.

Strategy Examples

BCV Strategy

class UrlBcv implements UrlStrategy
{
    protected $scraper;

    public function __construct($scraper)
    {
        $this->scraper = $scraper;
    }

    public function getValue(): float
    {
        $url = 'https://www.bcv.org.ve/';
        $value = $this->scraper->scrapeData($url, 'bcv');
        return $value;
    }
}

BDV uses a JSON API endpoint instead of web scraping, demonstrating the flexibility of the strategy pattern.

3. UrlProviderService

This service acts as a factory for strategy instances:

app/Services/UrlProviderService.php

public function getStrategy(string $identifier): UrlStrategy
{
    return match ($identifier) {
        'banplus' => new UrlBanplus(new ScraperService()),
        'bnc' => new UrlBnc(new ScraperService()),
        'bcv' => new UrlBcv(new ScraperService()),
        'bdv' => new UrlBdv(),
        default  => new UrlBdv(),
    };
}

4. FetchExchangeRates Command

The artisan command orchestrates the entire scraping process:

app/Console/Commands/FetchExchangeRates.php

protected $signature = 'rates:update';
protected $description = 'Consulta el BCV y otros bancos para actualizar tasas';

public function handle()
{
    $this->info('Iniciando actualización de tasas...');
    
    foreach ($this->banco as $banco) {
        try {
            $this->store($banco);
            $this->info("Banco {$banco} procesado.");
        } catch (\Exception $e) {
            $this->error("Error en {$banco}: " . $e->getMessage());
            Log::error("Fallo en cron para {$banco}: " . $e->getMessage());
        }
    }
    
    $this->info('Tasas actualizadas exitosamente.');
}

Error handling: If one bank fails, the command continues processing others.

Data Flow

1. Scheduled Trigger

   Laravel scheduler triggers rates:update command at configured times (11 times daily).

   routes/console.php

   $schedules = ['00:00', '00:30', '02:00', '03:00', '03:30', '04:00', 
                 '05:00', '05:30', '06:30', '07:00', '07:30'];
   
   foreach ($schedules as $time) {
       Schedule::command('rates:update')->dailyAt($time);
   }
   
2. Strategy Selection

   For each bank, the command gets the appropriate strategy from UrlProviderService.

3. Data Retrieval

   The strategy calls getValue() which either:

   • Scrapes HTML using ScraperService (BCV, Banplus, BNC)
   • Fetches JSON from API (BDV)
4. HTML Parsing

   For scraped sources, ScraperService uses CSS selectors to extract the rate:

   // BCV example
   private function parseBCVData($crawler)
   {
       $element = $crawler->filter('#dolar');
       $text = $element->text();
       
       if (preg_match('/USD\s+(.*)/', $text, $matches)) {
           if (preg_match_all('/[0-9]+,[0-9]+/', $matches[1], $coincidencias)) {
               $valor = trim($coincidencias[0][0]);
           }
       }
       
       return $this->cleanValue($valor);
   }
   
5. Data Normalization

   Raw values are cleaned and converted to float:

   private function cleanValue($value)
   {
       return (float) preg_replace('/[^0-9.]/', '', str_replace(',', '.', $value));
   }
   
6. Database Storage

   Rates are stored in the reference_records table:

   $record = new ReferenceRecord();
   $record->source = $banco;  // 'bcv', 'banplus', etc.
   $record->value = $value;    // 45.25
   $record->date = $today;     // '2026-03-04'
   $record->save();
   
7. Cache Invalidation

   A model observer automatically invalidates the cache when new records are saved:

   app/Observers/ReferenceObserver.php

   public function saved(ReferenceRecord $referenceRecord): void
   {
       Cache::forget('tasas_bancos_' . $referenceRecord->date);
   }
   

API Request Flow

When a client makes an API request:

1. Route Matching

   routes/api.php

   Route::get('/', [ScraperController::class, 'show']);
   Route::get('/info/{date}/{source?}', [ScraperController::class, 'getInfo']);
   
2. Cache Check

   Controller checks if cached data exists:

   app/Http/Controllers/Api/ScraperController.php

   public function show(Request $request)
   {
       $cacheKey = 'tasas_bancos_' . now()->toDateString();
       
       $records = Cache::remember($cacheKey, 3600, function () {
           return ReferenceRecord::where('date', now()->toDateString())
               ->get()
               ->keyBy('source');
       });
       
       if ($records->isEmpty()) {
           return response()->json(['message' => 'Tasas no disponibles aún'], 404);
       }
       
       return response()->json([
           'message' => 'Consulta exitosa',
           ...$records->map(fn($item) => [
               'value' => $item->value,
               'date' => $item->date
           ])->all()
       ]);
   }
   
3. Database Query (if cache miss)

   If cache doesn’t exist, query the database and cache the result for 1 hour (3600 seconds).

4. Response Formatting

   Data is transformed to a clean JSON structure with bank codes as keys.

Scraping Techniques

CSS Selector Targeting

Each bank uses specific selectors:

• BCV: #dolar - ID selector for the dollar rate element
• Banplus: .awb-news-ticker-link - Class selector for news ticker
• BNC: .ItemSpace - Class selector, then text filtering

Regular Expression Parsing

After selecting elements, regex extracts numeric values:

// Matches patterns like "45,25" or "45,30"
preg_match_all('/[0-9]+,[0-9]+/', $text, $matches)

Request Headers

Mimics a real browser to avoid being blocked:

'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9',
'Accept-Language' => 'es-ES,es;q=0.9,en;q=0.8',

Error Handling

Network Failures

HTTP errors return null, allowing the system to continue with other banks.

Parsing Failures

If a selector doesn’t match, an exception is thrown and logged, but the command continues.

Duplicate Prevention

Before scraping, the system checks if data already exists for today:

$exists = ReferenceRecord::where('source', $banco)
    ->where('date', $today)
    ->exists();

if ($exists) {
    $this->info("El registro para {$banco} ya existe hoy. Saltando...");
    return;
}

Performance Optimizations

1. Caching: 1-hour cache reduces database queries
2. Indexed queries: Date and source fields should be indexed
3. Lazy loading: Only scrapes when scheduler runs
4. Fail-fast: Invalid responses return early
5. Observer pattern: Automatic cache invalidation on data changes