Preparing TextTools scripts

Documentation for Developing Scripts in TextTools

📋 Table of Contents

1. Introduction
2. Basic Script Usage
3. Working with CSV and Tables
4. Mathematical Calculations
5. Practical Examples
6. Complex Processing Chains
7. Security and Best Practices
8. External Libraries
9. Documentation References

🎯 Introduction

TextTools provides a powerful mechanism for executing custom scripts for text processing. Scripts run in a secure, isolated environment with access to built-in functions and external libraries.

Available Global Objects:

• TextTools - Core library with over 100 functions
• arquero - Library for processing tabular data
• formulajs - Excel-like calculation functions
• slugify - Transliteration and creation of URL-friendly strings
• pluralize - Working with plural forms
• console - Safe logging

📝 Basic Script Usage

Script Structure

// Input text is available as the variable `input`
// The result must be assigned to the variable `output`
output = input;

// Example: Basic text cleaning
output = TextTools.stripHtml(output);
output = TextTools.removeExtraSpaces(output);
output = TextTools.changeTextCase(output, 'lower');

// The script will automatically return the value of output

Quick Start Script Template

// Script: Data Cleaning and Normalization
output = input;

// 1. Cleaning
output = TextTools.stripHtml(output);
output = TextTools.removeEmojis(output);
output = TextTools.removeControlChars(output);

// 2. Normalization
output = TextTools.removeExtraSpaces(output);
output = TextTools.normalizeUnicode(output, 'NFC');

// 3. Formatting
output = TextTools.wordWrap(output, 80);

📊 Working with CSV and Tables

Using Arquero for Complex Operations

Example 1: CSV Data Analysis

// Convert CSV to arquero table
output = input;
const table = csvToArqueroTable(output);

// Column statistics
const stats = table.rollup({
  count: arquero.op.count(),
  avg: arquero.op.mean('price'),
  sum: arquero.op.sum('quantity')
});

// Filtering data
const filtered = table.filter(arquero.escape(d => d.price > 100));

// Sorting
const sorted = filtered.orderby('price');

// Grouping
const grouped = table.groupby('category')
  .rollup({ total: arquero.op.sum('amount') });

// Convert back to text
output = arqueroTableToText(sorted, 'csv');

Example 2: Processing Log Data

output = input;
const table = csvToArqueroTable(output);

// Adding calculated columns
const enhanced = table.derive({
  profit: d => d.revenue - d.cost,
  margin: d => (d.revenue - d.cost) / d.revenue * 100
});

// Filter by condition
const profitable = enhanced.filter(
  arquero.escape(d => d.margin > 20)
);

// Aggregation by day
const daily = profitable
  .derive({ date: d => d.timestamp.substring(0, 10) })
  .groupby('date')
  .rollup({
    total_profit: arquero.op.sum('profit'),
    avg_margin: arquero.op.mean('margin'),
    count: arquero.op.count()
  });

output = arqueroTableToText(daily, 'markdown');

Example 3: Merging Multiple CSVs

// Assume input contains multiple CSVs separated by "---"
output = input;
const csvs = output.split('---');
const tables = csvs.map(csv => csvToArqueroTable(csv));

// Merge tables
let merged = tables[0];
for (let i = 1; i < tables.length; i++) {
  merged = merged.concat(tables[i]);
}

// Remove duplicates
merged = merged.dedupe();

// Sort by date
merged = merged.orderby('date', 'desc');

output = arqueroTableToText(merged, 'csv');

🔢 Mathematical Calculations

Using formulajs for Excel-like Calculations

Example 1: Financial Calculations

output = input;
const table = csvToArqueroTable(output);

// Add financial indicators
const calculated = table.derive({
  npv: d => formulajs.NPV(d.discount_rate, ...d.cash_flows),
  irr: d => formulajs.IRR(d.cash_flows),
  pmt: d => formulajs.PMT(d.rate, d.periods, d.pv),
  fv: d => formulajs.FV(d.rate, d.periods, d.payment, d.pv)
});

output = arqueroTableToText(calculated, 'csv');

Example 2: Statistical Analysis

output = input;
const numbers = TextTools.extractNumbers(output, 'list', true)
  .split('\n')
  .map(n => parseFloat(n))
  .filter(n => !isNaN(n));

if (numbers.length > 0) {
  const stats = {
    count: numbers.length,
    sum: formulajs.SUM(numbers),
    average: formulajs.AVERAGE(numbers),
    median: formulajs.MEDIAN(numbers),
    stdev: formulajs.STDEV.S(numbers),
    min: formulajs.MIN(numbers),
    max: formulajs.MAX(numbers)
  };
  
  output = JSON.stringify(stats, null, 2);
}

🛠️ Practical Examples

Common Replacements and Transformations

Example 1: Phone Number Normalization

output = input;

// Convert all phones to a single format
output = output.replace(/(\+7|8)[\s\-\(\)]*(\d{3})[\s\-\(\)]*(\d{3})[\s\-\(\)]*(\d{2})[\s\-\(\)]*(\d{2})/g, 
  '+7 ($2) $3-$4-$5');

// Remove extra characters
output = output.replace(/[^\d\s\+\-\(\)]/g, '');

Example 2: Splitting Complex Strings

output = input;

// Split by multiple delimiters
const lines = TextTools.splitLines(output, /[;,|]\s*/g, true);

// Clean each line
const cleaned = lines.map(line => 
  line.trim().replace(/\s+/g, ' ')
);

// Remove empty lines and duplicates
output = cleaned
  .filter((line, index, self) => 
    line && self.indexOf(line) === index
  )
  .join('\n');

Example 3: Date Format Conversion

output = input;

// Support various date formats
const datePatterns = [
  /(\d{2})\.(\d{2})\.(\d{4})/g,      // DD.MM.YYYY
  /(\d{4})-(\d{2})-(\d{2})/g,        // YYYY-MM-DD
  /(\d{1,2})\/(\d{1,2})\/(\d{4})/g,  // MM/DD/YYYY
  /(\d{4})(\d{2})(\d{2})/g          // YYYYMMDD
];

const replacements = [
  '$3-$2-$1',  // to ISO
  '$1-$2-$3',  // already ISO
  '$3-$1-$2',  // to ISO
  '$1-$2-$3'   // to ISO
];

datePatterns.forEach((pattern, index) => {
  output = output.replace(pattern, replacements[index]);
});

Processing Specific Formats

Example 1: Parsing Nginx Logs

output = input;

// Extract key fields from logs
const logRegex = /^(\S+) (\S+) (\S+) \[([^\]]+)\] "(\S+) ([^"]+) (\S+)" (\d+) (\d+) "([^"]*)" "([^"]*)"/gm;

const parsed = [];
let match;
while ((match = logRegex.exec(output)) !== null) {
  parsed.push({
    ip: match[1],
    time: match[4],
    method: match[5],
    url: match[6],
    status: match[8],
    size: match[9],
    referer: match[10],
    agent: match[11]
  });
}

// Convert to CSV
if (parsed.length > 0) {
  const headers = Object.keys(parsed[0]);
  const csv = [
    headers.join(','),
    ...parsed.map(row => headers.map(h => `"${row[h]}"`).join(','))
  ];
  output = csv.join('\n');
}

Example 2: Processing JSON API Responses

output = input;

try {
  // Extract JSON from text
  const jsonText = TextTools.extractJson(output);
  const data = JSON.parse(jsonText);
  
  // Flatten nested structures
  const flatten = (obj, prefix = '') => {
    return Object.keys(obj).reduce((acc, key) => {
      const pre = prefix.length ? prefix + '.' : '';
      if (typeof obj[key] === 'object' && obj[key] !== null) {
        Object.assign(acc, flatten(obj[key], pre + key));
      } else {
        acc[pre + key] = obj[key];
      }
      return acc;
    }, {});
  };
  
  if (Array.isArray(data)) {
    const flattened = data.map(item => flatten(item));
    output = TextTools.convertJsonTo(JSON.stringify(flattened), 'csv');
  } else {
    output = JSON.stringify(flatten(data), null, 2);
  }
} catch (error) {
  // Return original text on error
  console.error('JSON processing failed:', error.message);
}

🏗️ Complex Processing Chains

Data Pipeline for ETL Processes

Example 1: Complete Data Cleaning Pipeline

// Step 1: Extraction and Validation
function extractData(text) {
  return pipe(
    TextTools.stripHtml,
    TextTools.removeControlChars,
    TextTools.extractJson,
    json => {
      try {
        return JSON.parse(json);
      } catch {
        return { error: 'Invalid JSON' };
      }
    }
  )(text);
}

// Step 2: Transformation
function transformData(data) {
  if (data.error) return data;
  
  const table = arquero.from(data);
  
  return table
    .derive({
      timestamp: d => new Date(d.created_at).toISOString(),
      value_numeric: d => parseFloat(d.value) || 0,
      category_normalized: d => TextTools.changeTextCase(d.category, 'lower')
    })
    .filter(arquero.escape(d => d.value_numeric > 0))
    .select('timestamp', 'category_normalized', 'value_numeric')
    .objects();
}

// Step 3: Loading (Formatting)
function loadData(transformed) {
  return pipe(
    data => JSON.stringify(data),
    json => TextTools.convertJsonTo(json, 'csv'),
    csv => TextTools.sortCsvByColumn(csv, 0),
    csv => TextTools.csvToMarkdownTable(csv)
  )(transformed);
}

// Main script
output = input;
const extracted = extractData(output);
const transformed = transformData(extracted);
output = loadData(transformed);

Example 2: Text Data Analysis with Aggregation

output = input;

// Split into documents
const documents = output.split(/\n{3,}/);

// Process each document
const results = documents.map(doc => {
  // Cleaning
  const clean = pipe(
    TextTools.stripHtml,
    TextTools.removeExtraSpaces,
    t => TextTools.changeTextCase(t, 'lower')
  )(doc);
  
  // Analysis
  const stats = TextTools.countStats(clean);
  const keywords = TextTools.extractWithRegex(clean, /\b\w{5,}\b/g, '$&')
    .split('\n')
    .filter(k => k)
    .slice(0, 10);
  
  const entities = {
    emails: TextTools.extractEmails(clean).split('\n').filter(e => e),
    phones: TextTools.extractPhoneNumbers(clean).split('\n').filter(p => p),
    urls: TextTools.extractUrls(clean).split('\n').filter(u => u)
  };
  
  return {
    stats,
    keywords,
    entities,
    sentiment: formulajs.AVERAGE(
      ...clean.split(/[.!?]+/)
        .map(s => s.length)
        .filter(l => l > 0)
    ) // Simple "complexity" metric
  };
});

// Aggregate results
const summary = {
  total_documents: documents.length,
  avg_word_count: formulajs.AVERAGE(results.map(r => r.stats.words)),
  total_entities: {
    emails: formulajs.SUM(results.map(r => r.entities.emails.length)),
    phones: formulajs.SUM(results.map(r => r.entities.phones.length)),
    urls: formulajs.SUM(results.map(r => r.entities.urls.length))
  },
  top_keywords: Array.from(
    results.reduce((map, r) => {
      r.keywords.forEach(k => map.set(k, (map.get(k) || 0) + 1));
      return map;
    }, new Map())
  )
    .sort((a, b) => b[1] - a[1])
    .slice(0, 20)
    .map(([word, count]) => `${word} (${count})`)
};

output = JSON.stringify(summary, null, 2);

Recipes for Common Tasks

Recipe 1: Processing User Data

// Template for cleaning and normalizing user input
const userDataPipeline = pipe(
  // 1. Sanitization
  TextTools.stripHtml,
  TextTools.removeEmojis,
  TextTools.removeControlChars,
  
  // 2. Normalization
  t => TextTools.normalizeUnicode(t, 'NFC'),
  TextTools.removeExtraSpaces,
  
  // 3. Standardization
  t => TextTools.changeTextCase(t, 'title'),
  
  // 4. Validation
  t => {
    const emails = TextTools.extractEmails(t);
    const phones = TextTools.extractPhoneNumbers(t);
    return `Validated:\nEmails: ${emails.split('\n').length}\nPhones: ${phones.split('\n').length}\n\n${t}`;
  }
);

output = userDataPipeline(input);

Recipe 2: Preparing Data for Machine Learning

// Prepare text data for ML
const mlTextPreprocessor = pipe(
  // Cleaning
  TextTools.stripHtmlTags,
  TextTools.removeDiacritics,
  t => t.replace(/[^\w\sа-яА-ЯёЁ]/g, ' '),
  
  // Normalization
  TextTools.removeExtraSpaces,
  t => TextTools.changeTextCase(t, 'lower'),
  
  // Tokenization
  t => t.split(/\s+/).filter(word => word.length > 2).join(' '),
  
  // Deduplication of lines
  TextTools.removeDuplicateLines,
  
  // Length filtering
  t => t.split('\n')
    .filter(line => line.split(' ').length >= 5 && line.split(' ').length <= 100)
    .join('\n')
);

output = mlTextPreprocessor(input);

🔒 Security and Best Practices

Prohibited Constructs

The following patterns are prohibited in scripts:

• `eval()`, `Function()` - dynamic code execution
• `setTimeout()`, `setInterval()` - asynchronous operations
• `fetch()`, `XMLHttpRequest` - network requests
• DOM manipulation (`document`, `window`)
• Storage access (`localStorage`, `indexedDB`)
• System APIs (`process`, `require`, `import`)

Security Recommendations:

1. Validate input data inside the script
2. Limit complexity of computations
3. Use try-catch for error handling
4. Log operations via `console.log`
5. Test scripts on small data sets

Example of a Secure Script

try {
  output = input;
  
  // Validate input data
  if (output.length > 100000) {
    throw new Error('Input too large (max 100KB)');
  }
  
  // Safe processing
  const lines = output.split('\n');
  const processed = lines
    .slice(0, 1000) // Limit number of rows
    .map(line => {
      // Safe operations only
      const clean = TextTools.stripHtml(line);
      return TextTools.truncateText(clean, 500);
    });
  
  output = processed.join('\n');
  
} catch (error) {
  console.error('Script error:', error.message);
  output = `Error: ${error.message}\n\nOriginal input:\n${input}`;
}

📚 External Libraries

Arquero (Table Data Processing)

Official Documentation:

• GitHub Repository
• API Documentation
• Usage Examples

Key Capabilities:

• Filtering, sorting, grouping
• Aggregation and window functions
• Table joins (join, merge)
• Vectorized operations

FormulaJS (Excel Functions)

Official Documentation:

• GitHub Repository
• Function List
• Calculation Examples

Supported Categories:

• Financial functions (NPV, IRR, PMT, FV)
• Statistical (AVERAGE, MEDIAN, STDEV)
• Logical (IF, AND, OR, NOT)
• Text (TEXT, CONCAT, LEFT, RIGHT)
• Date/Time (DATE, NOW, DATEDIF)

🚀 Quick Start for Developers

Template for Creating New Scripts:

/**
 * Name: [Your Script Name]
 * Description: [Brief description of functionality]
 * Author: [Your Name]
 * Version: 1.0
 */

'use strict';

output = input;

try {
  // === CONFIGURATION ===
  const CONFIG = {
    maxInputSize: 100000, // 100KB
    maxRows: 10000,
    encoding: 'UTF-8'
  };
  
  // === VALIDATION ===
  if (output.length > CONFIG.maxInputSize) {
    throw new Error(`Input exceeds maximum size of ${CONFIG.maxInputSize} bytes`);
  }
  
  // === PROCESSING ===
  console.log('Starting processing...');
  
  // Your logic here
  // Use TextTools.*, arquero, formulajs
  
  // === RESULT ===
  console.log('Processing completed successfully');
  
} catch (error) {
  console.error('Script execution failed:', error);
  output = `❌ Error: ${error.message}\n\nDebug info:\n${JSON.stringify({
    inputLength: input.length,
    timestamp: new Date().toISOString(),
    error: error.message
  }, null, 2)}`;
}

// Return the result

Debugging Scripts:

// Use console.log for debugging
console.log('Input type:', typeof input);
console.log('Input length:', input.length);
console.log('First 100 chars:', input.substring(0, 100));

// Check library availability
console.log('TextTools available:', !!TextTools);
console.log('arquero available:', !!arquero);
console.log('formulajs available:', !!formulajs);

// Log intermediate results
const intermediate = TextTools.stripHtml(input);
console.log('After stripHtml:', intermediate.length);

Performance:

// Optimization for large data
const optimizeForLargeData = (text) => {
  // Process line by line, not all at once
  return text.split('\n')
    .map(line => {
      // Fast operations
      return line
        .replace(/<[^>]*>/g, '') // Faster than stripHtml for simple cases
        .trim()
        .replace(/\s+/g, ' ');
    })
    .filter(line => line.length > 0)
    .join('\n');
};

output = optimizeForLargeData(input);

Note: All scripts execute in an isolated browser environment. Your data never leaves your computer, ensuring maximum privacy and security.