Script for downloading all owned tracks in Wikiloc

Disclaimer

Jordi Ramot, Wikiloc’s creator, asked me to remove the script from GitHub, argumenting that it could add an additional load to already over stressed servers. I agreed, since I’m a avid Wikiloc user and I’d like to have this awesome and free service up and running.

Original post

You have tens or hundreds of track in Wikiloc. You didn’t store them locally on your computer. Now you want to download them. Going through the list of tracks in Wikiloc and downloading them individually is tiresome.

On the other hand, I wanted to practice my Ruby skills. The solution is a small script written in Ruby. You can download it from GitHub: https://github.com/mgryszko/wikiloc-downloader

It uses Mechanize and Nokogiri for the scraping part (you have to install the Mechanize gem (gem install mechanize).

The usage is simple:

Usage: download_all_tracks [options]
    -u USER                          email or user name
    -p PASSWORD

I hope this functionality will be implemented directly in Wikiloc – your data is yours, so you should be able to download it.

Android form conversion and validation with Option monad

What I want to do

  • Given a form in an Android activity or dialog, convert form values (java.lang.Strings) to domain classes and validate if they fulfill business constraints
  • Don’t validate if there is a conversion error
  • Chain validators
  • Perform complex validations (e.g. of two dependent fields)

Solution with Option monad

I encapsulated a form value coming from a visual component (it can be an EditText, Spinner, whatever) in the FormValue monadic type. This is a straight implementation of the Option monad with two subtypes:

  • Some containing the valid value
  • None containing the error description

See the source code below for the implementation:

public abstract class FormValue<T> {
    public static <T> FormValue<T> some(T value) {
        return new Some(value);
    }

    public static <T> FormValue<T> none(String error) {
        return new None(error);
    }

    private FormValue() {}

    public abstract boolean hasValue();
    public abstract T value();

    public abstract boolean hasError();
    public abstract String error();

    public abstract FormValue<T> validateWith(Validator<T> validator);

    public static class Some<T> extends FormValue<T> {
        private T value;

        private Some(T value) { this.value = value; }

        @Override
        public boolean hasValue() { return true; }
        @Override
        public T value() { return value; }

        @Override
        public boolean hasError() { return false; }
        @Override
        public String error() {
            throw new NotImplementedException("Not an error");
        }

        @Override
        public FormValue<T> validateWith(Validator<T> validator) {
            return validator.validate(value);
        }
    }

    public static class None<T> extends FormValue<T> {
        private String error;

        private None(String error) { this.error = error; }

        @Override
        public boolean hasValue() { return false; }
        @Override
        public T value() {
            throw new NotImplementedException("Not a value");
        }

        @Override
        public boolean hasError() { return true; }
        @Override
        public String error() { return error; }

        @Override
        public FormValue<T> validateWith(Validator<T> validator) {
            return this;
        }
    }
}

How does it work?

Conversion

First, we have to extract the raw value stored in a widget and convert it to a domain representation wrapped in FormValue. We do this with the ViewValueExtractor interface:

public interface ViewValueExtractor<T> {
    ViewValueExtractor<T> withMessage(String message);

    FormValue<T> extractValue();
}

Implementations of this interface depend on the widget. In the simplest form, it extracts a String value from a TextView doing no conversion (and thus not raising any errors):

public class StringExtractor implements ViewValueExtractor<String> {
    private TextView view;

    public StringExtractor(TextView view) { this.view = view; }

    public ViewValueExtractor<String> withMessage(String message) {
        return this;
    }

    public FormValue<String> extractValue() {
        return FormValue.some(view.getText().toString());
    }
}

Another example converts a text field to the SSN class, requiring AAA-GG-SSSS format:

public class SSN {
    // getters and setters omitted for brevity
    public int areaNumber;
    public int groupNumber;
    public int serialNumber;

    public SSN(int areaNumber, int groupNumber, int serialNumber) {
        this.areaNumber = areaNumber;
        this.groupNumber = groupNumber;
        this.serialNumber = serialNumber;
    }
}

public class SSNExtractor implements ViewValueExtractor<SSN> {
    private TextView view;
    private String message;

    public SSNExtractor(TextView view) {
        this.view = view;
    }

    public ViewValueExtractor<SSN> withMessage(String message) {
        this.message = message;
        return this;
    }

    public FormValue<SSN> extractValue() {
        String text = view.getText().toString();
        String[] ssnParts = text.split("-");
        if (ssnParts.length != 3) {
            return FormValue.none(message);
        }
        try {
            return FormValue.some(
                new SSN(
                    Integer.valueOf(ssnParts[0]), 
                    Integer.valueOf(ssnParts[1]), 
                    Integer.valueOf(ssnParts[2])
                )
            );
        } catch (NumberFormatException e) {
            return FormValue.none(message);
        }
    }
}

Generally speaking, you need to provide some way to pull out the value from a visual component and turn it into Some if the conversion was successful or None (with an error message).

Validation

We have already an object representing a value from our solution domain; now it’s time to validate it. We do it implementing a Validator:

public interface Validator<T> {
    Validator<T> withMessage(String message);

    FormValue<T> validate(T value);
}

It can be a generic validator:

public class GreaterThatOrEqualValidator<T extends Comparable<T>> implements Validator<T> {
    private T minValue;
    private String message;

    GreaterThatOrEqualValidator(T minValue) {
        this.minValue = minValue;
    }

    public Validator<T> withMessage(String message) {
        this.message = message;
        return this;
    }

    public FormValue<T> validate(T value) {
        if (value.compareTo(minValue) >= 0) {
            return FormValue.some(value);
        }

        return FormValue.none(message);
    }
}

Or a domain-specific validator, e.g. for our Social Security Number:

public class SSNValidator implements Validator<SSN> {
    private String message;

    public Validator<SSN> withMessage(String message) {
        this.message = message;
        return this;
    }

    public FormValue<SSN> validate(SSN value) {
        if (areaValid(value.areaNumber) 
            && groupValid(value.groupNumber) 
            && serialValid(value.serialNumber)) {
            return FormValue.some(value);
        }

        return FormValue.none(message);
    }

    private boolean areaValid(int areaNumber) {
        return areaNumber >= 0 && areaNumber <= 999;
    }

    private boolean groupValid(int groupNumber) {
        return groupNumber > 0 && groupNumber <= 99;
    }

    private boolean serialValid(int serialNumber) {
        return serialNumber > 0 && serialNumber <= 9999;
    }
}

If you go back to the implementation of FormValue.validateWith, you may notice that Validator.validate will be called only for Some value. In case of None (resulting e.g. from a wrong conversion), validateWith returns the same None object.

Complete example – conversion + validation

We convert the SSN field to SSN and check if it’s valid. In case of errors (during conversion or validation), an error message will be displayed in a Toast.

public class MonadicValidatorDemoActivity extends Activity {
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        findViewById(R.id.validate).setOnClickListener(
            new View.OnClickListener() {
                public void onClick(View v) {
                    validateForm();
                }
            }
        );
    }

    private void validateForm() {
        TextView ssnField = (TextView) findViewById(R.id.ssn);
        String wrongSSNFormatMsg = getString(R.string.wrongSSNFormat);
        String invalidSSNMsg = getString(R.string.invalidSSN);
        
        FormValue<SSN> ssn = new SSNExtractor(ssnField).withMessage(wrongSSNFormatMsg)
            .extractValue()
            .validateWith(new SSNValidator().withMessage(invalidSSNMsg));

        if (ssn.hasValue()) {
            Toast.makeText(this, "SSN: " + ssn.value(), Toast.LENGTH_LONG).show();
        } else {
            Toast.makeText(this, "Error: " + ssn.error(), Toast.LENGTH_LONG).show();
        }
    }
}

Chained and dependent validation

You can combine validators, adding more validateWith calls. Dependent validation is also possible:

  • either by implementing a validator factory accepting a FormValue and implementing a no-op or identity validator (NoOpValidator) – see the code example below
  • or by implementing a Validator verifying a FormValue, e.g. GreaterThatOrEqualValidator<T extends Comparable> implements Validator<FormValue>
private void validateForm() {
    // Field and message finders omitted
    FormValue<Date> startDate = new DateExtractor(startDateField).withMessage(invalidStartDateFormat)
        .extractValue()
        .validateWith(greaterThatOrEqual(today()).withMessage(startDateInThePast));

    FormValue<Date> endDate = new DateExtractor(endDateField).withMessage(invalidEndDateFormat)
        .extractValue()
        .validateWith(greaterThatOrEqual(today()).withMessage(endDateInThePast));
        .validateWith(greaterThatOrEqual(startDate).withMessage(startDateAfterEndDate));

    // Do something with form values and validation errors
}

private <T extends Comparable<T>> Validator<T> greaterThatOrEqual(T value) {
    return new GreaterThatOrEqualValidator<T>(value);
}

private <T extends Comparable<T>> Validator<T> greaterThatOrEqual(FormValue<T> value) {
    if (value.hasValue()) {
        return new GreaterThatOrEqualValidator<T>(value.value());
    }

    return new NoOpValidator<T>();
}
public class NoOpValidator<T> implements Validator<T> {
    private String message;

    @Override
    public Validator<T> withMessage(String message) {
        this.message = message;
        return this;
    }

    public FormValue<T> validate(T value) {
        return FormValue.some(value);
    }
}

Is it really a monad?

If a type wants to be a monad, it has to have:

  • unit operation that “wraps” a value with monad. It can be a constructor or a factory method. In our case FormValue.some represents the unit operation
  • bind operation transforming the monad into a next monad, exposing its internal value for a transformation function. FormValue.validateWith does this. Because Java doesn’t have first-class functions, we represent the validateWith operation argument as a method object – Validator

Furthermore, a monad must follow three monadic laws:

  1. identity – transforming to unit doesn’t change a monad
  2. m.bind { x -> unit(x) } ≡ m

    Previously mentioned NoOpValidator implements such a function:

    public FormValue<T> validate(T value) {
        return FormValue.some(value);
    }
    

    In case of Some, calling validateWith (bind method) with this Validator will return a new Some monad holding the same value. None‘s implementation returns always the same monad instance. Hence the first law is fulfilled.

  3. unit – unit must preserve the value inside the monad
  4. unit(x).bind(f) ≡ f(x)

    unit(x) corresponds to FormValue.some(x). Result of the execution of Some.validateWith is equivalent to the execution of the provided Validator:

    public FormValue<T> validateWith(Validator<T> validator) {
        return validator.validate(value);
    }
    
  5. associativity – order of monad composition doesn’t matter
  6. m.bind(f).bind(g) ≡ m.bind{ x -> f(x).bind(g) }

    Since None‘s validateWith returns always none (ignoring the bind function), it satisfies the associativity law.
    For Some it’s not so obvious. Let’s expand the left part of the theorem:

    some.validateWith(validator1).validateWith(validator2) => validator1.validate(value).validateWith(validator2)
    

    In order to implement the function in Java from the right side of the 3rd monadid law, we create an anonymous Validator:

    some.validateWith(
        new Validator<T>() {
            public FormValue<SSN> validate(SSN value) {
                validator1.validate(value).validateWith(validator2)
            }
        }
    );
    

    As you can see the result of the validation in the anonymous class is the same as of the expression expanded above.

    Conclusions

    Monads look abstract, esoteric, and, I admit, a little scary. Although from the functional programming domain, they can be implemented in imperative languages. They solve real-life problems (converstion and validation) in an elegant manner (we chained converters and validators with few ifs and no null checking).

Asynchronous tests with GPars (Osoco test gallery, part 4)

If you wrote multithreaded code in Groovy/Grails, I’m sure you stumbled upon the GPars library. It simplifies parallel processing – sometimes it’s as easy as wrapping a code fragment with a GPars closure, changing the iteration method (in our example from each to eachParallel) and passing the closure to a GPars pool. The library implementation handles the executor pool creation and adds new methods to collections.

Whenever I deliberately add multithreading, I always separate responsibilities (processing logic and concurrency handling). I wrap the single-threaded class with a new one, handing the parallel execution (creating a pool, managing threads, handling cancellation, etc.):

class MultithreadedProcessorService {
    ProcessorService processorService
    def threadPoolSize = ConfigurationHolder.config.threadPool.size

    void processMultithreaded(batches) {
        GParsPool.withPool(threadPoolSize) {
            batches.eachParallel { batch ->
                processorService.process(batch)
            }
        }
    }
}

class ProcessorService {
    void process(batch) {
        // some processing
    }
}

To test the multithreaded class you can you use Spock interactions. As opposed to Groovy and Grails mocks, they are thread-safe and you won’t get any weird and unstable results:

class MultithreadedProcessorServiceSpec extends UnitSpec {
    private MultithreadedProcessorService multithreadedProcessorService
    // Using Spock mocks because they are thread-safe
    private ProcessorService processorService = Mock(ProcessorService)

    def setup() {
        mockConfig('threadPool.size=2')
        multithreadedProcessorService = new MultithreadedProcessorService(
            processorService: processorService
        )
    }

    def 'processes batches using multiple threads'() {
        given:
        def batches = [[0, 1], [2, 3, 4], [5]]
        def processedBatches = markedAsNotProcessed(batches)

        when:
        multithreadedProcessorService.processMultithreaded(batches)

        then:
        batches.size() * processorService.process({ batch ->
            processedBatches[batch] = true
            batch in batches
        })
        assertAllProcessed(processedBatches)

    }

    private markedAsNotProcessed(batches) {
        batches.inject([:]) { processed, batch ->
            processed[batch] = false
            processed
        }
    }

    private void assertAllProcessed(batches) {
        assert batches*.value == [true] * batches.size()
    }
}

This post series present the best of Osoco tests – tests that were tricky or we are just proud of. You can find a runnable source code for this test and more in the Grails Test Gallery project shared on GitHub.

Grails base persistence specification (Osoco test gallery, part 3)

In our projects we include at least one integration test per domain class. We want to assure us that the GORM mapping is correct and the class is persistable in the target database.

The test plan is simple:

  • create a valid persistable sample of the domain object
  • save it
  • clear Hibernate first level cache (i.e. flush and clear current session)
  • retrieve the object once again and compare it with the previously saved object. They must be equal as by equals() but be different object references

We put the steps above in a base abstract specification. In concrete specifications we implement merely the factory method spawning a persistable object instance.

abstract class PersistenceSpec extends IntegrationSpec {
    SessionFactory sessionFactory

    protected abstract createPersistableDomainObject()
    
    def 'persistable domain object should be able to be saved and retrieved'() {
        given:
        def persistableDomainObject = createPersistableDomainObject()

        when:
        def savedDomainObject = persistableDomainObject.save()

        then:
        savedDomainObject.id != null

        when:
        clearFirstLevelCache()

        def retrievedDomainObject = persistableDomainObject.class.get(savedDomainObject.id)

        then:
        savedDomainObject == retrievedDomainObject
        !savedDomainObject.is(retrievedDomainObject)
    }

    private clearFirstLevelCache() {
        sessionFactory.currentSession.flush()
        sessionFactory.currentSession.clear()
    }
}

This post series present the best of Osoco tests – tests that were tricky or we are just proud of. You can find a runnable source code for this test and more in the Grails Test Gallery project shared on GitHub.

Grails equals and hashCode testing (Osoco test gallery, part 2)

If you work with GORM and your objects are held in collections, you should implement equals and hashCode methods. As everything, they should be thouroughly tested if they obey equals-hashCode contract. At Osoco we created a equals-hashcode-test Grails plugin that provides a base Spock specification for that.

In your equals and hashCode tests you simply have to extend the EqualsHashCodeSpec and provide at least two methods:

  • a factory method createDomainObjectToCompare that will create a fresh object for the comparison
  • a map of modified properties included in equals and hashCode
  • optionally a map of modified properties ignored in both methods

Additionally in case of inheritance, we check whether the equals method complies with the symmetry rule. We do it in an extra feature method added to the spec:

def 'equals relation must be symmetric with inheritance'() {
    given:
    def content = new Content()

    and:
    def book = new Book()

    expect:
    (content == book) == (book == content)
}

You may say, in Grails 2.0 and Groovy 1.8 there are @Equals and @HashCode AST transformations and you don’t need to write tests of them. I agree to some point. I think you can trust Groovy developers and you don’t have to test the methods for reflexivity, symmetry, transivity, consistency, and ‘non-nullity’. Anyway, you have to test the annotation configuration if it includes the properties that actually account for the logical equality between objects.

In the future releases of the plugin we will surely simplify the base specification and remove some checks but we will surely continue testing both methods (with our plugin it’s dirt-cheap!)


This post series present the best of Osoco tests – tests that were tricky or we are just proud of. You can find a runnable source code for this test and more in the Grails Test Gallery project shared on GitHub.

Follow

Get every new post delivered to your Inbox.

Join 318 other followers