Finish a/b test

[edenisn]

When we choose winner of a/b test and then click on link "make permanent" we confirm with, for example "Finish this experiment and assign all current and future participants to option B?". What I need to do if I don't want to assign all future participants to option B? I just want to close test without assign participants any more.

[phillbaker]

@edenisn if you want to end an experiment, by definition you're reducing the number of options to just one - users have to go into an alternative. I suppose the experiment file can also be deleted.

I'm not sure I understand the request?

[edenisn]

@phillbaker we don't want to delete experiment. We just want to finish the experiment and so that the number of participants to the winner option no longer increases.

[phillbaker]

Hm, still a little confused as to the desired behavior for users currently in the experiment and users entering the experiment. Can you clarify the behavior below:

• While experiment is active:
  • New users randomly assigned to alternatives (or assigned based on the strategy of the experiment)
  • Existing users continue to see the alternative they were shown
• When experiment is "finished" as requested in this issue:
  • New users _______ ?
  • Existing users continue to see the alternative they where shown?

[edenisn]

When experiment is "finished" as requested in this issue:

• New users still assign and increases option B. The number of participants still growing (If option B is a winner of experiment)
• Existing users continue to see the alternative they where shown? - Existing users continue to see option B (If option B is a winner of experiment)

[phillbaker]

Ah, I think this may be a dupe of #195 ?

[edenisn]

Yes, quite right

[edenisn]

In my opinion the best way out of this situation is to bring the statistics below the test at its end. For example:

Option A - 16 participants, 8 converted, 34%
Option B - 34 participants, 27 converted, 76%

Thus, we fix the statistical data at the end of the test

[phillbaker]

to bring the statistics below the test at its end

Can you explain this? I don't understand the suggestion.

I think we might be able to update the logic here:

vanity/lib/vanity/experiment/ab_test.rb

Lines 189 to 207 in 6e53223

	if @playground.collecting?
	if active?
	if enabled?
	return assignment_for_identity(request)
	else
	# Show the default if experiment is disabled.
	return default
	end
	else
	# If inactive, always show the outcome. Fallback to generation if one can't be found.
	index = connection.ab_get_outcome(@id) || alternative_for(identity)
	end
	else
	# If collecting=false, show the alternative, but don't track anything.
	identity = identity()
	@showing ||= {}
	@showing[identity] ||= alternative_for(identity)
	index = @showing[identity]
	end

to include whether the experiment has an outcome, and if so to not save additional choices to the DB. How does that sound?

Alternatively, experiments has a completed_at timestamp and participants have a created_at timestamp. Particpants added after the completed at could be ignored for statistics purposes.

[edenisn]

I mean that we simply can add more statistical information to /admin/vanity page when experiment finished. Add information with participants, converted and % at the end of the experiment.

By the way, your suggestion is very good. It would be fine if we don't save info with additional choices to DB after finished experiment.

Thanks @phillbaker