Laravel: Working a large number of tasks into a queue

I'm building a web application using Laravel 5 that creates links to the web application that when visited show a form for a student's progress report. These links are sent by the web application to an email of a contact at the institution the student attends in order for them to complete the progress report accessed by the link in the email.

The problem I face is when creating and sending links. I have some code which works fine with a couple of hundred students, however in real world use the application would be potentially creating and sending 3000 or so links at one time. The code I've written simply can't handle such a large number and the application crashes.

Although I'm more than welcome to other suggestions, I believe the answer to the problem is utilising queues. I have already used queues when sending email, but I would like to work some other parts of the code into queues but I'm a bit unsure how to do this!

Brief database schema

Student hasMany Link

Student hasMany InstitutionContact (limited to two by my application)

Link hasMany InstitutionContact (limited to two by my application)

Email manyToMany Link

What I am trying to accomplish

• Get all the Student's that require a new Link

• Create a Link for each Student

• Assign the Student's current InstitutionContacts to the Link's InstitutionContact

• Loop through all the newly created Links in order to group them together by shared InstitutionContacts - this is so an email is not sent per link (thus possibly sending too many emails to the same address), rather links should be grouped together by the same email/contact and sent together where applicable

• Loop through all the Links grouped by email/contact and:

  1. Send an email including the Link's info to the designated InstitutionContact's email address
  2. Write a copy of the Email to the database
  3. Join the Email created in the former step to the Link(s) that were sent in it

So the main challenge I'm facing is performing the aforementioned task with a large dataset. I have already considered creating and sending a Link one by one via a queue, however this wouldn't allow me to group all the Links together by contact/email. As the task wouldn't be performed regularly I would be open to consider performing the task as it is already with an increase in memory and time for the process, however I didn't have much success when attempting this using set_time_limit(0); and ini_set('memory_limit','1056M'); before sending any links.

Any help would be really appreciated, thank you if you read this far!

Code

app\Http\Controllers\LinkController.php

public function storeAndSendMass(Request $request)
{
    $this->validate($request, [
        'student_id' => 'required|array',
        'subject'    => 'required|max:255',
        'body'       => 'required|max:5000',
    ]);

    $studentIds = $request->get('student_id');
    $subject    = $request->get('subject');
    $body       = $request->get('body');

    $students = $this->student
        ->with('institutionContacts')
        ->whereIn('id', $studentIds)
        ->where('is_active', 1)
        ->get();

    // create link, see Link.php below for method
    $newLinks = $this->link->createActiveLink($students);

    // send link to student's contact(s), see LinkEmailer.php below for method
    $this->linkEmailer->send($newLinks, ['subject' => $subject, 'body' => $body], 'mass');

    // return
    return response()->json([
        'message' => 'Creating and sending links'
    ]);
}

app\Models\Link.php

public function createActiveLink($students)
{
    $links = [];

    foreach ($students as $student) {
        $newLink = $this->create([
            'token'          => $student->id, // automatically hashed
            'status'         => 'active',
            'sacb_refno'     => $student->sacb_refno,
            'course_title'   => $student->course_title,
            'university_id'  => $student->university_id,
            'student_id'     => $student->id,
            'institution_id' => $student->institution_id,
            'course_id'      => $student->course_id,
        ]);

        // $newLink->save();

        $studentContacts = $student->institutionContacts;

        if ($studentContacts) {

            foreach ($studentContacts as $studentContact) {

                $newLink->contacts()->create([
                    'type'                   => $studentContact->pivot->type,
                    'institution_contact_id' => $studentContact->pivot->institution_contact_id/*,
                    'link_id'                => $linkId,*/
                ]);

                $newLink->save();
            }

        }

        $links[] = $newLink->load('student');
    }

    return $links;
}

app\Emails\LinkEmailer.php

namespace App\Emails;

use App\Emails\EmailComposer;

class LinkEmailer
{
    protected $emailComposer;

    public function __construct(EmailComposer $emailComposer)
    {
        $this->emailComposer = $emailComposer;
    }

    public function send($links, $emailDetails, $emailType)
    {        
        $contactsAndLinks = $this->arrangeContactsToLinks($links);

        foreach ($contactsAndLinks as $linksAndContact) {

            $emailData = array_merge($linksAndContact, $emailDetails);

            // send/queue email
            \Mail::queue('emails/queued/reports', $emailData, function ($message) use ($emailData) {
                $message
                    ->to($emailData['email'], $emailData['formal_name'])
                    ->subject($emailData['subject']);
            });

            // compose email message, returns text of the email
            $emailMessage = $this->emailComposer->composeMessage($emailData);

            // // create Email
            $email = \App\Models\Email::create([
                'to'      => $emailData['email'],
                'from'    => 'report@domain.org',
                'subject' => $emailData['subject'],
                'body'    => $emailMessage,
                'type'    => $emailType,
                'user'    => $_SERVER['REMOTE_USER']
            ]);

            foreach ($linksAndContact['links'] as $link) {
                $link->emails()->attach($email->id);
            }
        }
    }

    // group links by contact
    public function arrangeContactsToLinks($links)
    {
        $contactsForLinks = [];
        $assigned         = false;
        $match            = false;

        foreach ($links as $link) { // 1, n

            if ($link->contacts) {

                foreach ($link->contacts as $contact) { // 1, 2

                    if ($contactsForLinks) {

                        $assigned = false;

                        foreach ($contactsForLinks as $key => $contactLink) { // n
                            // assign links to existing email in array
                            if ($contactLink['email'] === $contact->institutionContact->email) {
                                $match = false;

                                // check link hasn't already been included
                                foreach ($contactsForLinks[$key]['links'] as $assignedLink) {
                                    if ($assignedLink === $link) {
                                        $match = true;
                                    }
                                }

                                // if there was no match add to list of links
                                if (!$match) {
                                    $contactsForLinks[$key]['links'][] = $link->load('student');
                                    $assigned = true;
                                    break;
                                }
                            }
                        }

                        if (!$assigned) {
                            $contactsForLinks[] = [
                                'email'                 => $contact->institutionContact->email,
                                'formal_name'           => $contact->institutionContact->formal_name,
                                'requires_id'           => $contact->institutionContact->institution->requires_id,
                                'requires_course_title' => $contact->institutionContact->institution->requires_course_title,
                                'links'                 => [$link->load('student')],
                            ];
                        }
                    } else {
                        $contactsForLinks[] = [
                            'email'                 => $contact->institutionContact->email,
                            'formal_name'           => $contact->institutionContact->formal_name,
                            'requires_id'           => $contact->institutionContact->institution->requires_id,
                            'requires_course_title' => $contact->institutionContact->institution->requires_course_title,
                            'links'                 => [$link->load('student')],
                        ];
                    }
                }
            }
        }

        return $contactsForLinks;
    }
}

via Chebli Mohamed